Using a microcontroller and FreeRTOS for a data logger - Page 1
Forum section: electronics / general

author: SteveThackery
author_rank: Super Contributor
timestamp: August 01, 2024, 09:05:36 pm
content: PLEASE MOVE OR DELETE THIS IF IT'S IN THE WRONG FORUMColleagues!A few years ago, for my degree dissertation, I developed a data logger for monitoring various parameters of a motorcycle. Because of time constraints I decided to use a handful of Arduino Pro Minis as I knew them well. Each would monitor one or two inputs, and five times a second they would send their data to a "master" Arduino which was responsible for storing the data on an SD card for later analysis. The reason for dividing the workload across several Arduinos was due to their limited processing power. The peripheral Arduinos communicated with the master Arduino over I2C.Examples of the data gathered are: lambda sensor voltage; engine temperature; throttle position sensor (TPS); fuel injection pulse duration; manifold absolute pressure (MAP); road speed; engine rpm; brakes (applied or not); acceleration (from an accelerometer chip).These inputs were "scanned" five times per second, and this fact might be important for my question. So, take the fuel injection pulse duration, for example. It was not necessary or intended to log every pulse; rather, five times per second the duration of the most recent pulse was measured and reported back to the master Arduino. Another important point is that the fuel injection pulse duration is short (a few milliseconds) and needs measuring to tenths of a millisecond. Another Arduino monitored the TPS and reported its position when prompted by the master, and so on with the rest of the Arduinos.I always knew that using half a dozen Arduino Pro Minis all talking on an I2C bus was a horrendous bodge (although it worked well enough) and I always planned to implement the whole thing on a single, much more powerful, microcontroller. Each Arduino would be replaced by a thread (or task?) running in an RTOS.To my question... Firstly, from the above overview, is it the kind of application that could be met by a powerful microcontroller running FreeRTOS (for example)? If so, how frequently can I task switch the RTOS, bearing in mind that at least one of the parameters needs measuring to sub-millisecond resolution? I've read that an ESP32 Arduino running FreeRTOS task switches every millisecond, which would seem far too coarse to measure an input to one tenth of a millisecond.I realise I am probably missing some key concepts. I also realise I've skimmed over a lot of detail. But really I'm just hoping to get a handle on the feasibility of the project, and what the main issues would be regarding problems and solutions. Any thoughts or observations would be very much appreciated.

author: SteveThackery
author_rank: Super Contributor
timestamp: August 01, 2024, 09:22:55 pm
content: Just for a giggle, here is a picture of the Arduino-based prototype. Top right is a time-and-date clock module, and there is also an Arduino which I didn't mention, which did idle stabilisation.

author: SteveThackery
author_rank: Super Contributor
timestamp: August 01, 2024, 09:28:14 pm
content: Here is an extract showing how I compared the prototype architecture with the intended final architecture. I've just noticed a mistake in the Arduino numbering. These were illustrative only. In reality there were more Arduinos and proposed tasks.

author: pqass
author_rank: Super Contributor
timestamp: August 01, 2024, 10:46:43 pm
content: My first thoughts would be to apply analog signal conditioning to convert pulse-width or frequency sources to voltages. Then scale up/down the voltages via opamp/divider and read via single MCU ADC MUX. None of the sources move particularly fast (temp, pressure, RPM, etc.). You don't really need a real-time OS here.I'd setup a timer event (100Hz?) to collect one round of instantaneous voltages from all inputs, process a running average of each to suppress outliers, and append each to their queue/array of last-n averages. Occasional writes to the SDCARD are done in large-ish chunks to maximize life (suspending data collection while writing). If you're also managing multiplexed (say LED) displays, another timer event (60Hz * total digits) will be needed to update the current digit of the variable displayed. The single main thread can manage a command processor via serial.

author: brucehoult
author_rank: Super Contributor
timestamp: August 02, 2024, 01:21:44 am
content: Quote from: SteveThackery on August 01, 2024, 09:05:36 pmNooo. A tenth of a ms (100 µs, 1600 instructions) is a very long time even on an AVR. Write the software properly and one would be just fine.Not so long ago most 3D printers were controlled by AVR Arduinos, controlling the timing of individual stepper motor step pulses to X, Y, Z, and extruder stepper motors, running a PID control algorithm for the extruder heater, calculating acceleration profiles for the X and Y steppers, measuring the temperature, AND receiving and interpreting the G-code stream of commands, and often displaying status on an LCD.A Mega2560 was used in order to have enough pins, and enough code size, but it was no faster than the CPU in an Uno or Pro Mini.A better reason to use multiple MPUs would be to put them close to what they were measuring to reduce signal integrity issues, while having the logging communications over a channel with error detection and retry.You might also want/need a dedicated one to write the SD card, depending on how long writing a sector takes and whether it can be interrupted. TBH you'd be better off using a hardwired SPI flash chip, and then have a mode to dump its contents to an CD card (or out the UART or whatever) after the ride.But, yes, technology has moved on.For the last five years I'd say the go-to device for your needs would be the $25 or so Teensy 4.0 (or 4.1) board, with a 32 bit 600 MHz dual-issue Cortex-M7 (close to 900 MIPS and they're 32 bit MIPS), 1 MB RAM, a lot of GPIO (including analogue) pins. And it runs Arduino sketches with a fantastic OOTB experience -- I had mine running blinky in about 3 minutes flat.As an example of the performance, on my personal primes benchmark  a Teensy 4.0 took 43.5 seconds at its default 600 MHz, while a 20 MHz ATMega2560 took 13449.5 seconds, a factor of just over 300x.Ok, so that's 2019 tech.My go-to now is the $3 Milk-V Duo. It's got not 1 MB RAM but 64 MB, has a 700 MHz "microcontroller" core *plus* a 1 GHz (actually the default software runs it 850 MHz, but w/e) core running full Linux (typically Buildroot for the user-land, but you could put Debain/Ubuntu Server etc). It's got 26 GPIOs (only 2 ADC capable, sadly). And you can run Arduino sketches on the microcontroller core at the same time Linux is running on its core.https://milkv.io/docs/duo/overviewI don't know the exact times off-hand, but either the Teensy 4 or the Duo should be able to switch threads in an RTOS (or Linux for the Duo) in well under a µs let alone a ms.Heck, I'd think an AVR could switch threads in under 10 µs.https://hoult.org/primes.txt

author: NorthGuy
author_rank: Super Contributor
timestamp: August 02, 2024, 02:31:13 am
content: If an arduino can collect data from other arduinos, why cannot it collect the same data directly from sensors?

author: SiliconWizard
author_rank: Super Contributor
timestamp: August 02, 2024, 02:33:36 am
content: More so than speed, I'd say the insane idea of using something like a RTOS on a AVR chip is the very limited amount of RAM, rather than context switching time. But to each their own. If one absolutely wants to do it, doable it is.Yes, there are tons of much more powerful and even cheaper alternatives these days, even for those that want to keep using the Arduino framework.

author: ledtester
author_rank: Super Contributor
timestamp: August 02, 2024, 03:46:15 am
content: Quote from: SteveThackery on August 01, 2024, 09:05:36 pmI can see where interleaving the sensor access and SDcard writes could be kinda tricky on an Arduino. I think, though, you could still do all of the data collection with one Arduino and then use serial communication to pass it to another for writing to the SDcard. Have a look at the OpenLog project for an implementation of such a logger:https://github.com/sparkfun/OpenLogFor the fuel injection pulse length measurement from what you described this is how I would go about doing it:- Set up a pin change interrupt service routine for the GPIO pin- Inside the service routine, if the GPIO pin is currently HIGH, call millis() to get the current time and store it in the global variable pulse_start_time.- If the GPIO pin is LOW, compute millis()-pulse_start_time to get the duration of the pulse in milliseconds and store that in another global variable pulse_length. This variable will always reflect the length of the last pulse so your main loop can decide how often it wants to report it.There's a bunch of implementation details that you need to get right, but this approach avoids needing to poll the GPIO pin in a loop.Here's a pretty good tutorial on Arduino interrupt routines and covers pin change interrupts:https://dronebotworkshop.com/interrupts/

author: mianos
author_rank: Regular Contributor
timestamp: August 02, 2024, 04:34:35 am
content: Yes, the ESP runs a fork of FreeRTOS but that's not going to give you the ability to read all those sensors in real time.This is more in the domain of interrupts. There are modules for almost all the peripherals you mention that have read (past tense) trigger pins or direct triggers after some conditioning.If I was reading injector pulse width, I would just condition the signal, connect it to an interrupt and read them all. Then put this into a proper real time queue and chuck out the ones that I don't use.If there are too many pins you can use an io multiplexor and just read it to find out what pin changed in the interrupt and send that to the queue with the real time timestamp.

author: pcprogrammer
author_rank: Super Contributor
timestamp: August 02, 2024, 05:28:50 am
content: You don't need a RTOS for doing this.Use a micro with a couple of timers and fast adc's to do it all interrupt based and just run a main loop to do some processing on the data and write it to the SD card. Might even be doable on an AVR, but an ARM Cortex running on 40MHz should certainly be able to pull it off.The Raspberry pico (RP2040) could be a cheap option and allows doing some "task" in "hardware".An ESP32 running on 200MHz might even be overkill, but is also cheap enough.The Milk-V Duo brucehoult mentioned would definitely be overkill, but it is cheap and with Linux running on it makes writing to an SD card a simple job.

author: pcprogrammer
author_rank: Super Contributor
timestamp: August 02, 2024, 05:44:16 am
content: Quote from: brucehoult on August 02, 2024, 01:21:44 amA bit of a side track, but I wonder why the Milk-V Duo is rather expensive on Aliexpress or Amazon?https://nl.aliexpress.com/item/1005005699176591.htmlhttps://nl.aliexpress.com/item/1005006064411603.htmlOn AmazonDid not look at the shipping cost to France for the cheapest one I found, but the price difference in its base is big.https://arace.tech/products/milk-v-duo?variant=42555014512820

author: SiliconWizard
author_rank: Super Contributor
timestamp: August 02, 2024, 06:03:07 am
content: The lowest price for the Duo on Aliexpress is around $7 currently. It has always been higher than on Arace, which makes sense as Aliexpress is mainly resellers. They need to take their share too.Anything above $7-$8 (for the Duo, not the Duo-256M which obviously is more expensive) is to be avoided. Takes a bit of time to go through all offers to get those cheaper prices.Not talking about Amazon, which is obviously mostly people importing stuff from China and reselling there with a x3 to x4 markup.

author: brucehoult
author_rank: Super Contributor
timestamp: August 02, 2024, 06:35:23 am
content: Quote from: SiliconWizard on August 02, 2024, 06:03:07 amArace works for me, though even for just one Duo they want NZ$21 (US$12.50) shipping. But you can order 25 Duos for the same $21 shipping (then it goes to $26), so you're averaging US$3.84 shipped.Some of the Aliexpress probably have cheaper shipping.Anything above $7-$8 (for the Duo, not the Duo-256M which obviously is more expensive) is to be avoided. Takes a bit of time to go through all offers to get those cheaper prices.This one is showing me $2.67 for 1, with free shipping... They've sold quite a lot of them so probably ok.https://www.aliexpress.us/item/3256805512861839.htmlQuoteExactly.Ok, if you really need one tomorrow ... (and you're in the US)

author: brucehoult
author_rank: Super Contributor
timestamp: August 02, 2024, 06:48:28 am
content: Quote from: SiliconWizard on August 02, 2024, 02:33:36 amI dunno. If each thread is just a main function that doesn't really call anything else, but just waits on something and processes it and loops, then it might not be using any more stack/RAM than the 32 bytes of registers you have, or even less. It's not really any different to using an interrupt handler, except easier to write. If you're only using 30 bytes or even 100 bytes in each thread then you can fit quite a lot of them into even 2k bytes on an ATMega328. Not to mention that the ATMega2560 (Arduino Mega) has 8k of RAM.

author: peter-h
author_rank: Super Contributor
timestamp: August 02, 2024, 07:02:03 am
content: I've done a product which runs on a 32F417 and FreeRTOS runs at the usual Systick of 1kHz. Then a separate time interrupt runs various timers also at 1kHz (complicated reasons why those two are separate). But FR could run at 10kHz perfectly well, on this 168MHz CPU.At some £6 I would not bother with a slower CPU for anything unless it is high volume and has to be really cheap, or very low power, etc.Integrating FR is not hard. It is basically a standalone chunk of code, you make a call to initialise it, other calls to hang tasks onto it, and then a call to start it. It has to be hooked up to a 1kHz (etc) tick. It is super-reliable.One drawback of an RTOS is that it does need more RAM than doing things in a loop or state machine type of structure, because each task needs enough RAM for its own variables and stack, and since you need some fair margin for the stack, this margin gets duplicated by the # of tasks you have.

author: brucehoult
author_rank: Super Contributor
timestamp: August 02, 2024, 07:17:15 am
content: I just tried the following test (used by OpenBenchmarking / Phoronix) on the $3 64 MB Duo:https://github.com/detailyang/ipc_benchmarkCode:[root@milkv-duo]~/ipc_benchmark# ./run_tests.shposixq128 256 512 1024 204810MB/s 18MB/s 32MB/s 59MB/s 89MB/s81051msg/s 73967msg/s 65330msg/s 60416msg/s 45607msg/spipe128 256 512 1024 20488MB/s 14MB/s 28MB/s 47MB/s 81MB/s65461msg/s 57734msg/s 57439msg/s 48405msg/s 41419msg/sfifo128 256 512 1024 20488MB/s 16MB/s 28MB/s 48MB/s 82MB/s65198msg/s 65094msg/s 58310msg/s 49627msg/s 42043msg/ssocketpair128 256 512 1024 20487MB/s 14MB/s 26MB/s 49MB/s 85MB/s56833msg/s 57458msg/s 52297msg/s 50592msg/s 43704msg/suds128 256 512 1024 20487MB/s 14MB/s 25MB/s 50MB/s 83MB/s60991msg/s 55764msg/s 51126msg/s 51191msg/s 42537msg/stcp128 256 512 1024 20481MB/s 1MB/s 2MB/s 29MB/s 41MB/s4749msg/s 4604msg/s 4750msg/s 30117msg/s 21126msg/sshm128 256 512 1024 20486MB/s 13MB/s 26MB/s 46MB/s 89MB/s52358msg/s 51260msg/s 52298msg/s 47284msg/s 45452msg/sThe first five methods for IPC are all 12-17 MICRO seconds per 128 byte message.This is between Linux processes, not just threads in one process -- each test programfork()s a child and then communicates with it.The only changes I made to the github code were to useriscv64-linux-gnu-gccas the compiler and to add "-static" toCFLAGS,scp'd the binaries and scripts (well, ok .. the whole directory actually) to the Duo and ran./run_tests.shAll toooo easy.The VisionFive 2 runs the first four methods at 0.3 to 1.0 µs per message.

author: SiliconWizard
author_rank: Super Contributor
timestamp: August 02, 2024, 07:37:37 am
content: Quote from: brucehoult on August 02, 2024, 06:35:23 amYeah, that must be one of these first-time-buy prices, because for me the Duo on this page shows 11.75€ per 1 and 9.60€ for >= 3.

author: brucehoult
author_rank: Super Contributor
timestamp: August 02, 2024, 08:04:14 am
content: Quote from: SiliconWizard on August 02, 2024, 07:37:37 amYeah. I made sure I was logged in to my Ali account, but I haven't used *that* store before. And I went right to the "Charge my card NOW" screen and the price didn't change.It does say "Max. 1 pcs/shopper".You might not be seeing the cheap price, but probably a lot of people here would.

author: SteveThackery
author_rank: Super Contributor
timestamp: August 02, 2024, 08:09:59 am
content: Quote from: NorthGuy on August 02, 2024, 02:31:13 amThat sounds like a very good question. The answer - which may well be ridiculous nonsense - is as follows...The data from each of the peripheral Arduinos is very small, typically 16 bits, if I recall correctly. So sending it over I2C is very quick. On the other hand,gatheringthat data might take several milliseconds (timing the fuel injection pulse duration) or tens of milliseconds (timing the road speed pulse interval). That's why I wanted to offload those tasks to peripheral Arduinos.Having read the other comments, it seems increasingly clear that this was a misconception, because measuring the interval between two pulses doesn't tie up the processor for all that time. Having said that, measuring the fuel injection pulses to 100us seems like it might be hard while doing everything else as well.

author: pcprogrammer
author_rank: Super Contributor
timestamp: August 02, 2024, 08:42:58 am
content: Quote from: SiliconWizard on August 02, 2024, 07:37:37 amI see the same here. So they add to the price when you shop in Europe. Build in shipping cost probably.Quote from: brucehoult on August 02, 2024, 06:35:23 amIt most likely won't work out like that for me due to import duties since VAT is not included. I read their shipping policy and it is also a bit of a risk with them shifting liability for the shipping to you, so when it gets lost there is no money back guarantee.That is a nice thing of Aliexpress, they give some assurance, but you clearly pay for it too.

author: brucehoult
author_rank: Super Contributor
timestamp: August 02, 2024, 08:47:26 am
content: Quote from: SteveThackery on August 02, 2024, 08:09:59 amExactly.QuoteNot really.Just useattachInterrupt(digitalPinToInterrupt(pin), my_handler, CHANGE)to specify a small function to call whenever the value of an input pin changes. Then in that function callmicros()to get the current time and add it into some global variable or buffer that will be read and processed by the main program.Interrupt handlers should be short and return as quickly as possible.https://www.arduino.cc/reference/en/language/functions/external-interrupts/attachinterrupt/https://www.arduino.cc/reference/en/language/functions/time/micros/

author: brucehoult
author_rank: Super Contributor
timestamp: August 02, 2024, 08:55:56 am
content: Quote from: pcprogrammer on August 02, 2024, 08:42:58 amIt will work out like that for me. I've made several purchases at Arace.QuoteAliexpress are big enough (sell $80k+ of things to NZ a year) that they are in the system where they automatically add NZ GST to prices and forward it to the NZ government. Such shipments automatically fly through the border.Arace are small and under the radar so they don't add GST. Shipments to NZ valued less than NZ$1000 (US$600) to private individuals don't attract any taxes or duties or fees at the border, so they are released as soon as they determine the value.I'm sorry if you live in a less enlightened country or regional protectionist trade bloc. And the UK recently left one but then instituted their own stupid import charges! ID10Ts. What was even the point?

author: SiliconWizard
author_rank: Super Contributor
timestamp: August 02, 2024, 09:00:53 am
content: Yeah, there used to be a minimum value over here too for import taxes for private individuals, but the limit kept decreasing over the years. And now I think it's due from the first euro in the EU zone.

author: pcprogrammer
author_rank: Super Contributor
timestamp: August 02, 2024, 09:06:23 am
content: Quote from: SteveThackery on August 02, 2024, 08:09:59 amExamine all the different tasks that you have and think of other ways to implement them without some sort of polling. Measuring a pulse time can even be done completely in hardware with the more advanced timers, and only interrupt the processor after the end of the pulse. It might even be possible with the STM32 timers to only interrupt every so many pulses measured to reduce the load a bit further.It does require studying the reference manual and understanding all the possibilities of the hardware. Can be a good learning project.

author: SteveThackery
author_rank: Super Contributor
timestamp: August 02, 2024, 09:24:37 am
content: Firstly, I just want to say how thankful I am for all the brilliant replies. Collectively you have shown me that my basic assumption - that this is a job for an RTOS - is completely flawed. That's a great learning point for me and I really appreciate it.On the broader question, having shown that my data logger should be implemented differently, what sort of applicationwouldbe suitable for an RTOS?

author: Phil1977
author_rank: Frequent Contributor
timestamp: August 02, 2024, 09:27:08 am
content: I know the problem: A lot of sensors, from my experience especially these with UART-interfaces, have sometimes unexplainable timing issues and then it´s often helps to have one controller with a low-level-microcode (like an Arduino) to talk to them, collect the data in the RAM and send it via I2C to the logging system. I can completely understand the setup you have chosen for the engine-logger.Anyhow, there are several ways to do it better:- Use asynchronous buffer functions of the uC whereever possible. Think about asynchronous buffer ICs if you need more ports- Combine as many sensor readouts to one uC as it manages to read out the sensors serially. E.g. analogue signals are more or less gathered instantaneously, you can read out as many as the uC has ports. Some I2C readouts only take ms, some UART readouts can take hundreds of ms.- Use a buffer for writing or sending the logged data. Arduino with its small RAM is not very well suited for this, RP2040 or ESP32 are better.- Create timestamps at the first acquisition level- Think about some shared medium data transfer as CAN/ethernet if the logging system gets bigger.My list is definitely not complete and maybe outdated. I never professionally "learned" these things, I just write out of many years of hands-on-experience with various logging systems.

author: pcprogrammer
author_rank: Super Contributor
timestamp: August 02, 2024, 09:39:40 am
content: Quote from: SteveThackery on August 02, 2024, 09:24:37 amWhen you have a system that has different control loops that can be time shared, an RTOS makes it simpler to implement this.For instance a watering and feeding system for growing plants. You can have multiple PID controls running, that basically use the same code with different parameters.A real time operating system can be a very simple task switcher and not provide other support, but it can also be a full support operating system like Linux where timing is better guaranteed then on a bog standard Linux. Or something in between depending on your needs.

author: brucehoult
author_rank: Super Contributor
timestamp: August 02, 2024, 09:44:20 am
content: Quote from: SteveThackery on August 02, 2024, 09:24:37 am"RTOS" is a very grandiose term for what can be quite a simple thing.It's not so much the application, it's how lazy you want to be in not having to decompose and invert the logic of your application to a main loop, polling, and a set of interrupt handlers and interleave the logic of all the different processes and protocols together into one mess.

author: peter-h
author_rank: Super Contributor
timestamp: August 02, 2024, 09:50:14 am
content: An RTOS is really brilliant. You can write loads of tasks and write each one as if it owns the machine.This greatly simplifies the structure of the whole thing.Obviously it brings lots of gotchas e.g. if you just wait in a loop then (depending on priorities) you will hang the machine until the next tick, which slows things down, so you need to keep it co-operative by instead calling e.g. osDelay(1) or some such. And if you have tasks passing messages to each other then you can have a whole load of fun too, with deadlocks and such. But a lot of RTOS products don't do much of that, or do it in a simple way.Most products have code modules which interact little and when they do, via clear variables e.g. a user interface (keypad+LCD?) and some code which does real work. Perfect for an RTOS.And once you develop a basic RTOS based platform then you can re-use it for any number of entirely different products. I've done just that with my 32F417 based one. It also has ETH and USB but that's another topic (a huge amount of work). Very easy to spin off custom products. Before RTOS, every new product (I've designed dozens of uC based boxes since 1980) was mostly starting again.One still uses interrupts from things like UARTs even if there is an RTOS there.

author: NorthGuy
author_rank: Super Contributor
timestamp: August 02, 2024, 12:52:42 pm
content: Quote from: SteveThackery on August 02, 2024, 08:09:59 amNo. You can use interrupts - when there's an edge, normal execution stops, you record the timing of the edge and then return back to what you were doing.Moreover, MCUs have lots of different periphery modules. You may be able to delegate lots of functions to them - they may record the timing of the edges for you. To get a pulse width, all you need to do might be to subtract two values. When you select your MCU, you start from looking at the list of periphery and plan how you can use them to your benefit. In the end, the CPU may do very little - most of the job is done by periphery.

author: peter-h
author_rank: Super Contributor
timestamp: August 02, 2024, 01:03:01 pm
content: Yes. With clever use of timers one can even get pulse width measurement done for you. If one is cunning one can even use DMA to copy timer values into RAM...

author: SteveThackery
author_rank: Super Contributor
timestamp: August 02, 2024, 06:43:15 pm
content: I just want to say a big "thank you" to everyone who has contributed. I've learned loads and got some new insights into how to approach design challenges like this.Thank you again - I really appreciate it.

author: zilp
author_rank: Frequent Contributor
timestamp: August 02, 2024, 09:37:25 pm
content: As it seems noone has said it explicitly: The hardware solution for measuring pulses is something called the capture function of a timer, which will save the current counter value of the timer to a register on a signal edge, which you then can read out.You might want to just take the datasheet of some microcontroller and read it to learn what kind of stuff the hardware is capable of. In the case of the "arduino AVRs", the 16 bit timer has a capture unit, for example, but you'll find similar things in most microcontrollers. But if you can accept a bit more measurement jitter, using interrupt to capture the timer value can work just fine as well.

author: SteveThackery
author_rank: Super Contributor
timestamp: August 03, 2024, 11:45:03 am
content: Thanks, @zilp. You are right - I don't know enough about the features already available in the hardware. This sort of stuff is pretty much completely hidden when you are messing around with Arduinos.I have implemented a type of multi-tasking in Arduino by creating a number of 'C' functions - one for each task - and implementing them as state machines. Each is designed to do one simple thing (eg check the status of an I/O line, or a timer), switch to the next state, and return to the caller as quickly as possible. The Arduino loop() then comprises little more than a list of calls to each function, so they are all called in turn, repeatedly, at maximum speed.The secret to success is to keep the dwell time in each function as short as possible. I suppose it's a crude form of co-operative multi-tasking. It is from this place that I began wondering about doing multitasking properly, with something like FreeRTOS.I am pleased to learn that my data logger happens not to be a good project to implement on an RTOS. Combining the code from five Arduinos onto one, decently powerful, microcontroller will be an interesting task in itself, albeit not one I can use to learn about FreeRTOS.

author: NorthGuy
author_rank: Super Contributor
timestamp: August 03, 2024, 02:06:40 pm
content: Quote from: SteveThackery on August 03, 2024, 11:45:03 amTypically, the things you do in the main loop won't be time critical, so you don't worry much about the timing. If you have something time critical, you would rather create an interrupt for it.It's like in real life. If you need to give a medicine to your mother at exact time, you don't accomplish this by making all your tasks throughout the day very short. You accomplish this by setting an alarm clock and abandoning your routine tasks when the alarm clock rings.

author: gpr
author_rank: Contributor
timestamp: August 05, 2024, 11:55:35 am
content: Quote from: SteveThackery on August 03, 2024, 11:45:03 amI'm not sure how you've come to that conclusion.Of course there is a learning curve if you've never used something like FreeRTOS but already know Arduino, this is a valid point.But I don't see technical reasons to dismiss RTOS for this project.

author: shtirka
author_rank: Contributor
timestamp: August 05, 2024, 02:06:17 pm
content: Quote from: gpr on August 05, 2024, 11:55:35 amEven if there is no technical reasons to dismiss an RTOS, it doesnt mean you should stick it into all sorts of places: with OP's requirements on timing and limited functionality, an RTOS will do more damage than goodIlya

author: peter-h
author_rank: Super Contributor
timestamp: August 05, 2024, 02:44:25 pm
content: I wrote my own simple RTOS c. 1989 when I got deeply into a complicated Z280 based product. It was actually working without an RTOS (a main loop plus a 100Hz timer int plus comms ints plus some generous FIFOs built with an ASIC) but by the time I got around networking these boxes together (with a custom token-ring LAN I designed, Z180/85C30, 1553 PHY, etc - this was almost before ETH came in) it just got unworkable.Since then I've used an RTOS for everything nontrivial. My current project (last 5 years maybe, arm32) is under FreeRTOS which works great and actually of all the external stuff in this box (LWIP, MbedTLS, ST ETH, ST USB, etc) it has been the simplest thing and the biggest productivity enhancer.And one tries to re-use expertise, not design everything from scratch. If I now had to design some fancy digital clock I would just chop down my PCB and use that for it.

author: gpr
author_rank: Contributor
timestamp: August 05, 2024, 03:25:28 pm
content: Quote from: shtirka on August 05, 2024, 02:06:17 pmIt has been an interesting discussion with a few good suggestions and clarifications from people. And eventually OP concludes that this is not a good project for RTOS, which was a bit unexpected. I'm just wondering what was the logic behind that conclusion and what damage RTOS can do here. Probably I missed something important in the thread.

author: SteveThackery
author_rank: Super Contributor
timestamp: August 05, 2024, 03:40:42 pm
content: Quote from: gpr on August 05, 2024, 03:25:28 pmAs a newbie, it is perfectly possible that I have misunderstood the advice. If so, I'll change my mind again.The messages I'm picking up suggest that a stupidly fast task switching clock tick would be required to time event durations in software to a resolution of 100us. On the other hand, many micros have sophisticated timer features built in which could do a similar job with only a trivial software overhead.The other message I'm getting is that the time-critical event handling would be much more subject to an interrupt-based architecture.As I say, I might have misunderstood the comments in this thread, and I remain totally open to mew lessons and changing my mind where appropriate. Please - if you feel I've come to the wrong conclusion, please say so. I'd be delighted to continue this thread.

author: Phil1977
author_rank: Frequent Contributor
timestamp: August 05, 2024, 03:47:21 pm
content: Quote from: SteveThackery on August 05, 2024, 03:40:42 pmEspecially with a multitasking OS you should use the hardware timers and buffers where ever possible. And if you do so you can easily get good resolution.

author: dietert1
author_rank: Super Contributor
timestamp: August 05, 2024, 03:51:15 pm
content: Arduino is a little boring, isn't it?Nowadays we have very powerful MCUs for little money that easily run open source packages like FreeRTOS, LwIP or a USB stack. Every embedded developer should use those packages once in his/her life. One needs to take a closer look in order to know what those tools are and how to use them. FreeRTOS isn't just task switching. It is task switching based on queues, semaphores and timers. It works very well with interrupts and depending on what MCU you have, 100 usec may be 100 000 cycles, no problem at all.I am also integrating MbedTLS into a project. That STM32 MCU has a true random number generator based on analog hardware. No more pseudo random numbers!Regards, Dieter

author: SteveThackery
author_rank: Super Contributor
timestamp: August 05, 2024, 05:47:50 pm
content: Quote from: dietert1 on August 05, 2024, 03:51:15 pmI am keen to learn how to use an RTOS, and FreeRTOS does seem very suitable for microcontrollers.Quote from: dietert1 on August 05, 2024, 03:51:15 pmMy first adventure into an RTOS was the mBed environment, but I soon realised all was not well. The development UI was pretty rubbish, and I was soon asking on the forums about whether it was abandon-ware. Lo and behold, my fears came true, and now it really is officially abandon-ware (sorry, "deprecated").

author: gpr
author_rank: Contributor
timestamp: August 05, 2024, 06:15:48 pm
content: Quote from: SteveThackery on August 05, 2024, 03:40:42 pmYou can measure duration with hardware timer, either purely in hardware which will just give you pulse width in some register, or with some help from interrupts, when interrupts can be triggered on pulse edges for example. I doubt you can find some MCU nowdays without any support for this.Then why bother about tick clock? It is just a frequency how often scheduler decides which task to run next. Set it to 100Hz and forget about it, until you have a good reason to change it.QuoteIt depends what alternative you compare with interrupt-based architecture. Probably you mean polling, when you check some condition periodically with some delay, or even without any delay in a busy loop. Both approaches are valid, interrupts are generally have better latency. But interrupts don't come from nowhere, something should trigger it, for example:sensor toggles dedicated pin, interrupt is triggered by GPIO (pin peripheral), you know sensor has something to say, you read data from sensori2c peripheral (for example) detects i2c activity, triggers interrupt, you read the data from sensor via i2cDMA controller (configured beforehand for peripheral, i2c for example) triggers an interrupt, saying "I've completed reading data from i2c, data is ready in the memory you configured for it"About FreeRTOS (and probably any other embedded RTOS), there is a common misunderstanding that it limits what you can do in you code. No, if you use FreeRTOS doesn't mean you cannot use interrups or anything else. You still have unlimited full access to all your hardware, maybe except of a couple of system interrupts or a timer used by rtos for its own purposes. You can even continue enjoying your favorite state machine loop running in a separate task, concurrently with other tasks you might have. In that sense FreeRTOS adds you a possiblity to use multitasking with inter-task communication without any big disadvantages.If you are working on something hard realtime with tight timing requirements (not in this case), there can be some subtleties. But otherwise FreeRTOS is a very good tool to have in your toolbox.As you can see, this arduino project can be transformed to something really cool where you can learn a lot.

author: shtirka
author_rank: Contributor
timestamp: August 05, 2024, 06:48:31 pm
content: Quote from: gpr on August 05, 2024, 03:25:28 pmThe big difficulty with an RTOS in this context is that it would only primarily be used for a relatively simple task of logging - which not only defeats the whole point of an RTOS (usually it is used for use when more than one "logical" (to us humans) task needs doing, and also the effective throughput may well be well be well below the effort necessary to have an rtos in place for the task of logging in this context), it also adds quite a bit of complexity to an application (which in this case would defeat the point of an rtos simply for the purpose of making a smart logger). RTOS would make sense in a case when you add additional requitements to the logging application or if you restrict the timing or the effective throughput, but as it stands an RTOS would scare the heck out of the OP to begin with - youre trying to REDUCE complexity of an application, not add to it lolIlya

author: gpr
author_rank: Contributor
timestamp: August 05, 2024, 06:52:57 pm
content: Quote from: SteveThackery on August 05, 2024, 05:47:50 pmAlright, I see a bit of difficulty with FreeRTOS here. mBed has harwdare abstraction, you can use it without any hardware knowledge. Simply put, it has "I2C" class which you use, pass some parameters, and it just works. FreeRTOS doesn't have any built-in peripheral support (unless you use some third-party libraries, or FreeRTOS-based SDK from espressif for example), it is just a scheduler which switches tasks and allows them to communicate with each other.Personally, I prefer to use hardware directly without any libraries, maybe just CMSIS headers to use C structs instead of hardware addresses. This might sound complicated, but once you read datasheet and understand how some periprheral works, I find it much easier than HAL library.There are other choices if you want built-in peripheral support: espressif SDK, zephyr (which I don't reccomend as a first choice though), a bunch of python-based enviroments. Other people can give you better advice here.

author: peter-h
author_rank: Super Contributor
timestamp: August 06, 2024, 11:43:19 am
content: QuoteThat will seriously bite somebody in the bum one day.What do they do about /CS timing for example?Also, RTOS support for hardware devices is obviously tricky. Hardware is obviously not thread-safeSo how do you do that? One has to mutex each device, and then you (depending on detail) need to re-issue initialisation to it if the current one is for a different process, etc. I've done all that and it does work but there is a lot to know about what goes on behind the scenes.

author: Siwastaja
author_rank: Super Contributor
timestamp: August 06, 2024, 12:09:44 pm
content: Most I2C sensors decouple the actual sampling from reading it out, anyway - either by implementing a FIFO which can hold many samples, or at very least buffering one sample (so fixed FIFO length of 1). This means that exact timing is non-critical as long as you read out the sample before the next one arrives*, which is pretty easy if sample rate is low.*) if that next sample is relevant. If you just want data at say 10Hz and sensor samples internally at 100Hz, it's usually not a problem to let the data register overrun and lose 90% of the samples, althoughaliasingmight be an issue if input values are changing faster than your final sample rate.Same is true for sampling analog signals - triggering ADC sampling can be done accurately and jitter-free e.g. by a timer, so that readout timing is irrelevant (as long as it's before the next sample).DMA, if available, extends this even further, so that you don't need to process between every two samples, but you can collect many samples in memory automagically for later batch processing. Although, even if youcould, it's another question if youshould; on microcontrollers, timing is usually easy anyway, so if you just guarantee doing all the work between two samples, then job done, no need to think about how to process them in batch. Again, like in real life: if you work at a box factory assembling cardboard boxes that come in on a conveyor, it's so much easier if you are quick and dedicated enough to finish each before the next one arrives. Failing that, youcancollect them, but that brings new management tasks and requires area to store half-finished boxes; and eventually the conveyor needs to stop so that you can finish your work. And microcontrollers do not need coffee breaks, so simple "do it fully once it arrives" works very well.But pure datalogger is actually one of the examples where DMA and batch processing probably does make sense, and life easier - significantly decoupling sample generation from storage. This is useful because on datalogger, latency usually is irrelevant - data is looked at hours, not nanoseconds, later. DMA into memory allows leeway for stuff like SD card / filesystem latency / timing uncertainties.Lacking DMA, you basically do the same by reading a single value in an interrupt handler, writing it in a memory buffer. At trivially slow data rates, this isn't a problem.And RTOS solves none of this. If you use RTOS, you have exact same choices to make: whether to use interrupts for data collection, or configure DMA for the same job and then manage the DMA in interrupts.With an OS, you have a third option of creating abstract concept of "threads" that all busy loop looking at the inputs and let the scheduler to swap between these threads, but it's highly inefficient way to do it, and probably not even useful from code clarity viewpoint. I suggestevent-drivenprogramming paradigm, and it maps pretty well on hardware interrupts, especially on modern microcontrollers that implement pre-emptive (meaning: interrupts can interrupt other interrupts in priority order) interrupt controllers on hardware. If you choose this paradigm, then you end up using very few of RTOS features. Maybe some FIFOs / locking primitives offered by OS. Which you can do without the OS pretty easily.