Arduino IDE internal structure - Page 1
Forum section: electronics / general

author: rhb
author_rank: Super Contributor
timestamp: July 19, 2019, 03:38:15 pm
content: Can someone point me to a description of how the internals are organized? I'm looking for a short detailed description of things like linker search paths, preprocessor symbol definitions, where libraries and header files are located, file and directory naming conventions, etc.All I could find with google was novice level stuff. During my active career I had to sort out the structure of million line code bases which at the time I took responsibility for them could not be compiled. So I started with several thousand files on disk. In some cases I did not even have a list of the programs, so I had to start by identifying the main programs among all those files. Just getting a clean compile with all warnings set took 6-9 months.The Arduino IDE makes me nervous because it hides a lot of the critical details from the user. That's appropriate for its intended user base, but poses significant extra work as it means I might have to read all the Arduino IDE source to understand how to fix some libraries.I just received the USA Cal Club 2 kit. I *think* I've fixed the bug in the Tempduino code which appears to have been caused by the Arduino IDE not doing what the syntax of the void loop() function implies per the C++ standard on my bookshelf. Hence my concern. Was the bug a gcc bug or an Arduino bug? In this case, it appears that initialization of automatic variables was *not* being performed each time loop() was called.In the process of reading the Tempduino code I saw a large number of latent bugs in some libraries which accompanied it. Most especially failure to test return codes.Thanks,Reg

author: rstofer
author_rank: Super Contributor
timestamp: July 19, 2019, 04:17:03 pm
content: It gets pretty deep but the compiler is in:C:\Program Files (x86)\Arduino\hardware\tools\avr\binMost of what you want is in:C:\Program Files (x86)\Arduino\hardware\tools\avrYou can actually find the search paths that the compiler uses byC:\Program Files (x86)\Arduino\hardware\tools\avr\bin>avr-c++ --helpand then learning which options will print what you want to know. -dumpspecs will print out more than you ever want to know.If you just wander around the Arduino directory tree you can find the answers to most of your questions.

author: Mechatrommer
author_rank: Super Contributor
timestamp: July 19, 2019, 04:39:52 pm
content: Quote from: rhb on July 19, 2019, 03:38:15 pmits just normal structure that complies to C/C++ rules? just look inside the arduino folder its all in there. the structure is pretty organized imho from looking on the surface compared to most of the codes in public github etc.Quote from: rhb on July 19, 2019, 03:38:15 pmwhat do you mean by novice and not novice? the not novice is to disassemble the linker (gcc)? or the arduino.exe? i'd rather be a novice..Quote from: rhb on July 19, 2019, 03:38:15 pmits not that difficult compared to other libraries that i've tried to understand. you may start with "..\Arduino**\hardware\arduino\cores\arduino\main.cpp" thats where main() { setup() ... loop() } is... and then just follow the rhythm (#include and functions call) from there. thats why i stick with WinXP and its search engine, excels in this kind of job (search files *.c/cpp/h/hpp with keywords ie function name).Quote from: rhb on July 19, 2019, 03:38:15 pmwhat do you expect? trying to understand a middle level complexity structure of most codes/libraries in github or public internet is a nightmare, zero documentation... doxigen is next to useless, imho. this arduino as it meant to be... simple...Quote from: rhb on July 19, 2019, 03:38:15 pmnot testing return codes is not necessarily a bug. we need to trace the (route or code execution) possibilities from start to the end if we want to decide if its a bug or not.. sometime it is sometime its not (bug not in the route) what we really need is a software for "code organizer" or "tree or relational view" to look back and forth quickly and effectively between functions/files/classes/definitions etc without losing the current reality. its been discussed before but the suggested SW never hit to my liking... i have in mind but dont have time anymore to develop such software, if i really get serious i'll just hit and open my diy "Library Manager" SW, very very less than ideal, but it can show something a bit such as file dependency. ymmv.

author: rhb
author_rank: Super Contributor
timestamp: July 19, 2019, 04:40:43 pm
content: Thanks. That ought to take care of most of it and the rest should be pretty simple. It would be nice if people had a better grasp of the importance of communicating with other programmers, but, alas, they do not.The Gnu crowd keep changing the rules just because they can which gets *really* annoying if you're firmly entrenched in the Bell Labs Unix tradition and understand that changing the command line options breaks things and causes huge headaches for admins in enterprise scale environments. Especially if you need to restore a file some senior manager deleted by mistake and discover that you don't have backups because some program quit setting the exit code correctly and the script that some admin wrote 10 years ago is misled into thinking that everything worked as intended.If it were not for Cygwin, I would not even attempt to work on software under Windows. I looked at some Windows native code once and promptly ran away screaming.

author: andersm
author_rank: Super Contributor
timestamp: July 19, 2019, 04:49:27 pm
content: Quote from: rhb on July 19, 2019, 03:38:15 pmCan you give an example? Because an elementary bug that would break almost all code is not very likely.

author: rhb
author_rank: Super Contributor
timestamp: July 19, 2019, 05:10:14 pm
content: Quote from: Mechatrommer on July 19, 2019, 04:39:52 pmNot checking return codes is a latent bug waiting to bite the user. I've had to fix hundreds of them. Tracing the execution paths is NP hard. The problem generally goes under the rubric of "Turing's halting problem" in the computer science literature.My introduction to Unix, C and X11 was a 5000 line graphics code written by Rock Ottolini at Stanford. It had two comments which read /* handling special cases */. I used the standard Unix utilities to build caller-callee tables and all manner of other structure analysis from scratch in a couple of weeks.But that program had all the files in a single directory with a Makefile, so the system dependencies were minimal. My intern task was to fix a byte order issue in the X11 based graphics program which I completed in 6 weeks.For the benefit of @andersmIfCode:while( true) {int a = 0;// some stuff}does not set a to zero *every* time it runs it will break all code. It appears that the Arduino IDE has an issue doing that properly because the bug I am working on was:Code:void loop(){double a = 0;float out;for( i=0; i<max; i++ ){a += foo();}out = a / max;// some stuff}which was overflowing because a was *not* being reset to zero as it should. All I did was to explcitly set a = 0 prior to the loop instead of relying on the compiler to do it.Q.E.DReg

author: Nusa
author_rank: Super Contributor
timestamp: July 19, 2019, 05:54:55 pm
content: Quote from: rhb on July 19, 2019, 05:10:14 pmRelying on the compiler to do it would be a programmer error. It's not a compiler problem, unless you're talking about global variables. The language does NOT initialize simple* local variables by default. Their value is indeterminate until assigned a value.*Of course any type with a constructor defines its own initialization, or lack thereof.

author: andersm
author_rank: Super Contributor
timestamp: July 19, 2019, 06:20:59 pm
content: Quote from: rhb on July 19, 2019, 05:10:14 pmIf you mean you added the "a = 0" assignment, then yeah. POD ("plain old data") variables with automatic storage duration are not initialized to any value by default. That's how C and C++ have always worked.https://en.cppreference.com/w/cpp/language/default_initialization

author: rstofer
author_rank: Super Contributor
timestamp: July 19, 2019, 06:40:41 pm
content: Right, automatic variables are NOT initialized by default in either C or C++. Global or static variables are initialized ONCE to 0 by default. Local variables are allocated on the stack and are not initialized. These variables need to be initialized in code. The loop() allocates any local variable on the stack so they default to garbage. The following code does what is expected:Code:void setup() {Serial.begin(9600);}void loop() {int i = 3;Serial.print(i);i+= 1;Serial.print(i);}Each time the loop() function is called, integer i is initialized to 3, printed, updated to 4, printed and loop() returns to the caller. The output is exactly as expected ...3,4,3,4,3,4...There are at least two levels of libraries being used in the Arduino environment: Libraries specific to Arduino hardware and the Arduino infrastructure and the libraries from avr-libc which allow the GCC compiler to build code for almost any AVRs.It is certainly possible to blow off the Arduino libraries and program the same hardware using just any old IDE plus the avr-gcc toolchain and the avr-libc libraries along with avr-dude to program the device. Maybe toss in a Makefile here and there. There is no reason in the world to be limited to the Arduino way of doing things.Levels of trust:Top: GCCThen: avr-libcThen: Arduino librariesFinally:User code

author: rhb
author_rank: Super Contributor
timestamp: July 19, 2019, 06:43:49 pm
content: Automatic variables *are* supposed to be initialized if it is done in the declaration. But I have not checked the fine print in the standards to see if that applies *every* time the function is called.Right now I have to get the cal club GPIB working.

author: rstofer
author_rank: Super Contributor
timestamp: July 19, 2019, 06:46:16 pm
content: Quote from: rhb on July 19, 2019, 05:10:14 pmloop() is 'called' by another loop hidden in the Arduino code. Since loop() is just a normal called function, local variables are allocated on the stack and not initialized by default. Not a bug, not a mistake, just the way C and C++ have always worked.

author: cv007
author_rank: Frequent Contributor
timestamp: July 19, 2019, 07:19:33 pm
content: I'm not an arduino user, but have used arduino for esp32 briefly.Turn on verbose in preferences. Compile. In the output find the avr-size command line (will be at the end, so easy to find). Copy the whole line. Go to command prompt, paste that line, replace avr-size -A with avr-objdump -S , then read the disassembled code (redirect to a file to make it easier, open in text editor).Find the main for( ;; ) loop somewhere- the loop() may end up inline. You will where those vars are initialized, and where the loop jumps back to.Just looking at what I think is the dht library (google search to github), it appears the library will read only after 2 seconds elapsed from last read unless you provide the 'force=true' for humidity( provide 1 argument- true), and also for temperature (which also will require specifying the S arg which is a bool for C or F, so provide true, true or false, true).https://github.com/adafruit/DHT-sensor-library/blob/master/DHT.cppThat may not be what you are having a problem with, just thought maybe if you are not seeing updates as fast as you think should be, that could be a cause. (I am mind reading here of course, and assume your code example is trying to parallel the tempduino code).edit-and this is what I found for the tempduino code (which lead me to the dht code)-https://github.com/TechUnboxed/Tempduino/blob/master/tempduino3/tempduino3.ino

author: Mechatrommer
author_rank: Super Contributor
timestamp: July 19, 2019, 08:11:57 pm
content: Quote from: rhb on July 19, 2019, 06:43:49 pmi think local variables with assigned values should be initialized with that values every time the function is called, when function exits, its local variables should go out of context (terminated or freed), and when called again, local variables will be allocated and initialized again and so on. so its a surprise if they are not re-initialized, maybe something else going on that needs investigation. maybe the variable is declared as static from behind or from some setting? its been a while sine i play C/C++ but some links will provide the point such as...https://stackoverflow.com/questions/42534977/c-declaration-and-initialization-variables-inside-while-loops

author: rstofer
author_rank: Super Contributor
timestamp: July 19, 2019, 08:35:15 pm
content: For embedded programs, there may be no purpose served in testing error returns. What can you do about them? Sure, there are many occasions where you might shut a process down but there are others where the error just isn't that important or perhaps it simply isn't feasible to shut something down.It's a judgement call. If you can't do anything about the error, there's no point in worrying (testing) about it.If there is some kind of error output device (stderr) then maybe a message is about all you can do. Deeply embedded processors may not have that feature.

author: rhb
author_rank: Super Contributor
timestamp: July 19, 2019, 08:40:43 pm
content: OK Here are the initialization rules.The first page is from the C standard, the next 2 pages are from the C++ standard.As for testing return codes, the programmer is supposed to determine what should be done if a call returns an error. Not just blindly assume that everything is OK. The user doesn't determine what to do. The programmer does. Failing to test retuirn codes in embedded software has *very* serious consequences. It gets people killed!Ask the families of people killed in a 737 MAX how they feel about errors in embedded software.

author: rstofer
author_rank: Super Contributor
timestamp: July 19, 2019, 08:40:55 pm
content: Quote from: Mechatrommer on July 19, 2019, 08:11:57 pmThat's exactly what happens and you can see this in the code I posted above. if 'i' is not initialized in the declaration, it MIGHT be zero but it is probable that it is any other value. I just tried removing the '=3' part and it just so happened that the value was 0. But you can't count on it because it really isn't initialized by the language.It would be fun to look at the assembly language output with and without the initializer. Just to see what GCC thinks...

author: rstofer
author_rank: Super Contributor
timestamp: July 19, 2019, 08:44:28 pm
content: Quote from: rhb on July 19, 2019, 08:40:43 pmBullet 5 of the first document says it all.

author: rstofer
author_rank: Super Contributor
timestamp: July 19, 2019, 09:07:01 pm
content: Quote from: rhb on July 19, 2019, 08:40:43 pmWe don't fly airplanes with Arduinos. We don't develop life-safety applications with open source software (I hope!). But if the manufacturer farms out the programming to people who don't know how airplanes work, it won't matter what tools they use.In the nuclear business we had similar issues. Once a program was certified correct, the code and the hardware it ran on were locked in place for the duration. That gets problematic as hardware falls off the end-of-life but that's the way it works. Nobody wants to go through the certification process just for giggles.The Apollo Guidance Computer is an example of programming excellence:https://newatlas.com/apollo-11-guidance-computer/59766/

author: mikerj
author_rank: Super Contributor
timestamp: July 19, 2019, 09:35:37 pm
content: Quote from: rhb on July 19, 2019, 08:40:43 pmYou sure enjoy your hyperbole. Failing to check a return code in an Arduino project is much more likely to either have no consequences at all, or it could make someones flashy LED program misbehave. The chances of a bug in an Arduino program causing death is somewhere between slim and none, and slim's out of town.

author: Mechatrommer
author_rank: Super Contributor
timestamp: July 19, 2019, 09:43:36 pm
content: Quote from: rhb on July 19, 2019, 08:40:43 pmmaybe the original programmer had tested all conditions and decided everything should be ok and removed error checking code? is what you assumed "blindly assume". until proven otherwise. have you checked the code and found possible bug? and its predicted misbehaviour?Quote from: rhb on July 19, 2019, 08:40:43 pmit depends... it may gets people killed, it may not. it may has *very* serious consequences, it may not, depending on situation.Quote from: rhb on July 19, 2019, 08:40:43 pmthe answer will always be "bad". but dont try to put one situation to another consequence. compromise can be made in one situation where in other situation it cant. ask anybody how they feel if someone kills a person, the answer will always be bad. but does that mean, someone kills an insect is also bad? another example... there's a muslim group terrorist attack in a bar killing few people, thats bad! conclusion, every muslim is a terrorist... plain ridiculous

author: rhb
author_rank: Super Contributor
timestamp: July 19, 2019, 09:44:13 pm
content: Software should be written correctly no matter what it runs on whether that is a supercomputer with a petabyte of DRAM or an embedded system with 128 bytes of SRAM and 2 KB of flash. It is the responsibility of the programmer to attend to all those details. I have a single standard of programming, the very best I know how to do. Others are free to do as they please.Shortly after I started using C and Unix I wrote as an amusement a small program which passed variables between functions by exploiting the layout of the stack. That was 30 years ago.The MIPS R2000 based DECstation and the Sparcstation 1 had just appeared. I spent a *lot* of time explaining to very senior scientists with PhDs from the very best schools that section 8.3.5 of the FORTRAN 77 standard says "upon return from a function or subroutine, NAMED COMMON may be undefined". My copy of the standard is MIA at the moment, but there is more verbiage about the scope of variables which are local to a function or subroutine which is closely related.Although I have never found supporting documentation, I am quite certain that that language was inserted in the standard at the behest of Burroughs so they could use automatic variable allocation in the FORTRAN 77 compiler for the 5000 series which was the first stack based computer.But I spent a lot of time telling people who came to me for help to add SAVE statements to their code.@rstofer provided a basic answer to my initial question. If someone knows of a document with more details about the structure of the Arduino IDE, please send me a PM. But at the moment I need to get my arms around EZGPIB, calibrate my DMMs and a few of the voltage references and get the kit in the mail to the next person. I feel very guilty every day I delay putting it in the post.I'll be happy to discuss this stuff later, but for now I must refrain from further participation in this thread.Have Fun!Reg

author: cv007
author_rank: Frequent Contributor
timestamp: July 19, 2019, 11:16:13 pm
content: simple test to show uninitialized var-Code:void loop() {uint8_t a;SREG = a;}//this is the loop- the stack ( a ) is previously setup but not initialized, so you get whatever value happens to be at Y+23// 1c4: 8f 89 ldd r24, Y+23 ; 0x17// 1c6: 8f bf out 0x3f, r24 ; 63but do this-Code:void loop(){uint8_t a= { SREG };SREG = a;}and you will get the a (on the stack) initialized to 0 every time through (and a is set to the SREG value). I will not post to objdump asm code, but it is as I describe.The avr compiler seems to sometimes use r1 (0) for an uninitialized var that has no value assigned (yet), but its not anything to rely on. I would imagine each compiler may act differently. The arm gcc compiler seemed to ignore using the uninitialized var at all when running a couple tests. Maybe it makes a difference what kind of storage the var uses- register/stack, and maybe the compiler writers eventually end up in a grey area.K&R 1.10 - Because automatic variables come and go with function invocation, they do not retain their values from one call to the next and must be explicitly set upon each entry. If they are not set, they will contain garbage.QuoteYou also have 'float out;' which is also uninitialized, but in this case harmless. For reliable code, just set the initial value.If you turn on all warnings in arduino preferences, you would see something like-Quotewhich would have pointed out your uninitialized vars as potentially being a problem (hence the warning, but not an error). I wouldn't ignore any warnings, even if 'harmless' in a specific instance (you simply cannot keep track of which ones are harmless, so make them all go away).

author: donotdespisethesnake
author_rank: Super Contributor
timestamp: July 20, 2019, 09:37:16 am
content: Quote from: rhb on July 19, 2019, 03:38:15 pmOh yeah, the "Tempduino" code, everyone knows about that. I keep a copy of it right here. And the well-known "Arduino IDE" C++ compiler. Yeah, that's a thing!Wait, no, I've no idea WTF you are talking about. What I am getting is that you are a programming genius but nevertheless got butt hurt by a bug in someone else's code. (You of course have never written bad code in your whole life).Quote from: rhb on July 19, 2019, 04:40:43 pmHow about you start first?

author: Nusa
author_rank: Super Contributor
timestamp: July 20, 2019, 06:08:38 pm
content: Use of terminology suffers here. The IDE is not the compiler. It's also not the libraries, although in this case it includes library tools.IDE = Integrated Development Environment. Basically a fancy editor. They call command line tools for the actual compiling, linking, and crossloading. The Arduino IDE is no exception. With a little research, you can set up any of the more generalized IDE's to do the same thing. Or you can do the whole thing at the command line, either by hand or with makefiles. You're on your own making makefiles, of course. You can use the IDE verbose output to determine exactly what command lines it's actually using.Command line tools are distributed in hardware/tools/avr/bin, including avr-gcc and avrdude, among others. They take the usual --help tag. Everything you need is there.If you don't like the setup/loop construction, just write your own int main() function to replace the weakly bound one provided:Code:/*main.cpp - Main loop for Arduino sketchesCopyright (c) 2005-2013 Arduino Team. All right reserved.This library is free software; you can redistribute it and/ormodify it under the terms of the GNU Lesser General PublicLicense as published by the Free Software Foundation; eitherversion 2.1 of the License, or (at your option) any later version.This library is distributed in the hope that it will be useful,but WITHOUT ANY WARRANTY; without even the implied warranty ofMERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNULesser General Public License for more details.You should have received a copy of the GNU Lesser General PublicLicense along with this library; if not, write to the Free SoftwareFoundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA*/#include <Arduino.h>// Declared weak in Arduino.h to allow user redefinitions.int atexit(void (* /*func*/ )()) { return 0; }// Weak empty variant initialization function.// May be redefined by variant files.void initVariant() __attribute__((weak));void initVariant() { }void setupUSB() __attribute__((weak));void setupUSB() { }int main(void){init();initVariant();#if defined(USBCON)USBDevice.attach();#endifsetup();for (;;) {loop();if (serialEventRun) serialEventRun();}return 0;}

author: NorthGuy
author_rank: Super Contributor
timestamp: July 20, 2019, 11:14:39 pm
content: If you sayCode:double a = 0;the compiler certainly should initialize it. If it was declared "static", this should be done once. Otherwise, this should be done every time the function starts - as the variable is on the stack and doesn't even exist between function calls.GCC does exactly that.I suspect this is some sort of user error because the code posted doesn't compile and thus cannot be actual Arduino code.

author: ledtester
author_rank: Super Contributor
timestamp: July 21, 2019, 04:56:35 am
content: The Arduino team has released "arduino-cli" -- a command line interface to the Arduino tool chain:https://github.com/arduino/arduino-cliI haven't tried it yet, but there is a "--debug" flag which is used to debug the CLI itself, and I'm sure it will show how the gcc compiler tools are called.

author: Rick Law
author_rank: Super Contributor
timestamp: July 22, 2019, 11:46:37 pm
content: Quote from: rhb on July 19, 2019, 03:38:15 pmComing from Greenleaf/Microsoft C (and Visual C++) background, the organization of include ".h" files to me seem very similar except the lack of a real "make" file controlled by the end user. Also, unfamiliar to me was the multiple copies of similar header files apparently for different hardware but not well described anywhere until you look inside the content of those include files.After initially digging into those includes, I conclude that the drop down selections (like what CPU and what Arduino board type, etc.) merely pre-define some #define names and include the right header. I largely left it alone as something that I really don't need to know how it does it as long as it works. Should I ever need to really know, (if I recalled correctly), Arduino IDE does have an option to be verbose about what it is doing - to the point of keeping all the temp files so looking into those temp files can tell me what the #define names expansions becames and what went on each step of the process.The part the IDE I found most annoying (and initially confused the daylight out of me) - The IDE "make" is annoying in the short cuts it took in the spirit of making it simple but instead made it confusing:I am still using V1.6.x, newer ones might have fixed this problem. It automatically include all the .c/.cpp/.ino files in your project directory. If I copied currentVersion.c to versionYesterdayAsBackup.c and left it in the same project folder, it would automatically compile both and link both into the project which of course causes a ton of compile/link errors. Since this is done behind the scenes, I have no control of it. I would prefer to have a drop down box or something that specify any additional files I wish to compile/link into this project instead of just grabbing everything in my project folder chain.

author: Mechatrommer
author_rank: Super Contributor
timestamp: July 23, 2019, 01:59:03 am
content: Quote from: Rick Law on July 22, 2019, 11:46:37 pmdittto!... one easy way to know which branch to compile without having to look at each single line is by removing the suspected unused code (files and directories) from the search path and compile the project. if success, that chunk of code is not used, vice versa...