STM32 DMA FIFO bursts and timing tolerances. - Page 1
Forum section: electronics / general

author: paulca
author_rank: Super Contributor
timestamp: January 17, 2023, 01:45:40 pm
content: I getting fed up with circular array buffers. To be honest they cause more problems than they solve.STM32 DMA supports multi-buffer mode where you can specify two distinct unrelated buffers instead of halving one buffer. However the STM32 HAL I2S driver does not support the use of this. I put it on the back burner to fix that by reimplementing my own I2S driver by borrowing and butchering the HAL code.In the interim I discovered another technique which might work as well.DMA Burst Mode and FIFOs. The idea I'm hoping to achieve is DMA'ing a buffer to the I2S peripheral's FIFO. Hoping that when the last DMA transfer completes into the FIFO the TX_Cmplt interrupt will fire and I will have just enough time to point it to it's new buffer before the FIFO runs out.It would seem however the FIFO is only 16 octets in size. So even a 16bit I2S stream only has 8 samples (4 stereo pairs). At 48K that's about 100us which should be fine. EDIT: Not sure on that number now. However there are other delays in the DMA/I2S peripheral process pipeline, end to end, the DMA cores themselves have FIFOs etc. As Peter pointed out in a post when a DMA transfer reports being complete may not actually mean the peripheral has completely finished the last transaction, just that it's read for a new one.The reason I'm posting, is, I'm obviously just discovering this technique and I'm giving you guys the opportunity to say "Nope, don't go there, it's won't work." or "Why not just do this instead?"Why don't I like basic circular buffers?1. They leave open the error condition of looping the same buffer over and over which creates dangerous audio output that can actually damage hearing.2. Handling multiple streams requires memcpy'ing buffers all the time to "get out of the way" of the DMA pointers.3. Managing buffer halves and flipping pointers is just so "flint stones"4. I NEED to change buffers. I need to adjust buffer lengths. I need fall back/fail safe "silent" error conditions.5. I have plenty of memory (1Mb) and plenty of power 450Mhz, it's time to bring out some of the "big iron" toys to the party.EDIT6: Circular buffers on ins, processing and output forces my timing into being 1 buffer long at each buffer transfer. With the asynchronous "pooled" buffers there can be multiple buffers for the same stream in play in the system.Here's my current approach: (coded, not tested yet)

author: DavidAlfa
author_rank: Super Contributor
timestamp: January 17, 2023, 02:06:37 pm
content: Why don't you simply use the half-transfer isr?

author: paulca
author_rank: Super Contributor
timestamp: January 17, 2023, 02:15:47 pm
content: Quote from: DavidAlfa on January 17, 2023, 02:06:37 pmBut do what in it?If you mean the half-transfer on the circular buffers. I need more than 2 halves and I need to change the halves at will.If you mean the half-transfer on the DMA to FIFO transfer. Hmm. It certainly gives me plenty of time, but maybe too much time. I mean what can I do at "half transfer", I can't change the buffer it's transferring at that point. I could delay for 1/2 a buffer so I'm hot looping on the DMA Busy flag with a buffer ready, but that won't work when there are 2, 3 or 4 receiving I2S streams. I can't block. EDIT: I "can" poll select loop though... hmmm....Maybe you mean something else.EDIT: If I use 192 byte buffers, but only fill 96 bytes, leaving the rest zeros. That gives the failsafe of silent failure. However, it might also give me (with the FIFOs) help, enough time to stop and restart the DMA stream... but as I type that, I know it won't work the DMA will already have started to transfer 0s and if you interrupt the I2S it resets it's FIFOs to 0.

author: DavidAlfa
author_rank: Super Contributor
timestamp: January 17, 2023, 05:38:21 pm
content: It's a circular buffer of ex. 1KB, so you get a dma isr every 512bytes, you have another 512bytes left to fill that half, giving plenty of time.96+96bytes might be too small, it all depends on the I2S rate.

author: paulca
author_rank: Super Contributor
timestamp: January 20, 2023, 04:26:43 pm
content: I am aware of how circular buffers work and how 1960s they are. Especially unbounded ones. My situation is just too complex for basic "ping pong", I need multi-buffer.I got the code I wrote (to abstract the audio buffers into a buffer pool) running tonight. It got far, far, further than I expected (for about 300 lines of fresh, never run code).After a few whoopsies with pointer deference and byteSize vs. word size it failed exactly where it was expected to... the Mix down, as I only have 1 I2S input running. The idle one never provides a buffer so it never mixes, perfect error case capture.However, I identified where the design is lacking in that regard and I don't need anything as medieval as a timer. I just carry the cascade through from the output I2S. When it needs data that is the call for the cascade to work backwards to the Mixer where the buffers decouple. In LL terms when the output I2S TX Complete fires I give it the next waiting pre-pared DMA buffer and immediately flag the mixer to mix "right now" the next one with whatever it's got, no delay. Input streams will quickly align to the output cadence. I hope.I am having to add latency by having "now" buffers and "next" buffers at the decoupling point, but that fine. The input streams have as many buffers as is in their pool to carry on out of sync and realign later.The trickier part I expect is making 100% sure buffers are returned when used and not accidentally leaked. A fixed amount of buffers are allocated on start up. Admin will monitor the total free and also monitor that with several metrics and check for consistency. Gets vs. Frees vs. Total available etc. Thread-safing it and prioritising interrupts to make that thread-safing practical will be a challenge.Note. The lower level coupling to I2S is isolated in one place. If it turns out I don't have enough time to restart the DMA transfer before the I2S FIFO expires, I have a plan B and a plan CI can make it work.It also technically allows me to completely decouple these buffer streams from I2S and use SPI instead if I choose.Neat trick. Instead of assigning buffers to "NULL" to denote they are unavailable, pointing them to a static ZERO array means any mistakes/crashes should play silence.Other bonuses. A single buffer is never shared between components concurrently. Buffer crashes and warp overs cannot occur. No DMA is reading from a bit of memory being written to. The only concurrency scope is the bufferpool pointers and their pass chain from GetBuffer to FreeBuffer is kept "sane".

author: Siwastaja
author_rank: Super Contributor
timestamp: January 20, 2023, 07:24:00 pm
content: 100us is MASSIVE amount of time to do something as simple as buffer swapping and DMA reconfiguration. 450MHz? That's 45000 clock cycles. You can probablycreatethe whole buffer (e.g., a software synthesizer) during that time.ISR entry, DMA reconfiguration where you just make a few checks and swap pointers, then exit the ISR, is maybe something like 50 clock cycles, i.e. 100ns or so.STM32 DMA supports automatic buffer swapping (our double buffer mode), but with such easy timing constraints, it doesn't matter.You can also just simply not use DMA at all and directly write into the I2S FIFO in an ISR handler every 1/fs or so. Heck, given Cortex-M7 at 400MHz, I have written a software DC/DC control loop which runs at 250kHz (including actual calculations, not just data copy), and an SDLC protocol input stage (with bit destuffer AND continuous CRC calculation) which runs at 500kHz. In interrupts.All options are open with so much excess CPU processing power available.The things you are working with are only problematic on desktop PC in userland applications. On MCU, you have interrupt latencies within tens of clock cycles, guaranteed, with nearly no jitter.

author: NorthGuy
author_rank: Super Contributor
timestamp: January 20, 2023, 09:57:51 pm
content: Another word for "circular buffer" is "queue". You queue your items on one end. And you pick up items in the same order from the other end as you please. It's really hard to find a better implementation for a queue other than a circular buffer.Your frustration with circular buffers is because you create a queue for bytes, but then you try to use it for blocks. This, of course, won't work well, but is easily fixed.Why don't you create a circular buffer for blocks? For example, each elements of the circular buffer will be the pointer to a data block.When you want to put a block into the circular buffer, you allocate memory for the block, you fill your block in, then you put the pointer to the newly created block into your queue.At the reading end, when DMA is done with a block, you de-allocate the block, then retrieve the pointer to the next block from the queue and give it to DMA to process.To allocate blocks, if you don't have a heap, you can create your own allocator. The simplest is a set of N buffers (each capable of holding the longest block) accompanied with one word of information - 0 means the buffer is empty, (x > 0) means the buffer holds a block of length x.

author: paulca
author_rank: Super Contributor
timestamp: January 21, 2023, 11:29:59 am
content: Good to know on the power. I am just not familiar and comfortable with what performance I have yet. So I'm taking measures I might not need to take. Note, an F411 will mix 2 I2S channels, even without DMA, I know I have got that prototype working ages ago. What it won't do however is run 2 shelf and 3 or 4 peak filters without DSP BiQuad optimisation as well as down mix 2 or 3 I2S. Including calculating the coeficients on the fly given user input.And... my architecture requires 2 of these 'buses'. 2 distinct mix and EQ stages. If I can still get a single H7 to do all that without hassle, then I suppose I best try and figure out how to maximise the number of I2S IO I can get from the SAI blocks etc.Circular DMA buffers with interrupts memcopying the buffer halves into the next stage would work, it would just be wasteful. It also has way more fiddly low level concurrency issues.The main issue I still don't see anyone appreciating is how do you mix 2, 3, 4 buffers of I2S together? Ultimately all the buffers need to exist at that time to mix them. However there is no way to synchronise the streams, or at least no easy way. I can start them as two adjacent lines of code, but they are slaves, when they start and stop and how they align is upto the master. You should, surely be able to see that if a buffer can slip by up to a buffer length, there is no way to synchronise within 1 buffer span. Something is going to trample over something.With my approach that doesn't matter. The input will get itself a new buffer, the mixer will run without a buffer (when I fix it) and if a buffer it early or late the concurrency colision scope is that of the pointers, no memcpy from LIVE buffers to protect. The pointer either points to the new buffer or it doesn't at the exact moment in time the mixer runs. A memcpy getting interrupted by an I2S stream writing into that buffer half and all hell breaks loose. Making a memcopy atomic sounds dangerous, a pretty long time to be blocked on a single core. If my design the software hangs up, the buffers run out and the stream stops. It doesn't blast me with a 1Khz looped sample or a wall of pink noise.It remains to be seen how often this occurs in the wild. Those brute force buffer drops will only be tolerable if they happen very infrequently. As everything will be running off different clocks, although the same rate. There will be slippage. I just hope it's not that often or I will need to start buffer padding and truncating at the frame level rather than the 1ms buffer level.

author: paulca
author_rank: Super Contributor
timestamp: January 21, 2023, 11:44:39 am
content: The dragon I can hear breathing in it's lair however is the I2S input/output formats.It would seem, as I'm using pre-build PCB modules I am not going to get the kind of choice and control over those as I would have liked. It looks as though, as least two components will not play nice at 16bit. One of them puts it on a 32bit frame which might be ok, I can handle both 16bit/16bit and 16/32bit by discarding the other 16bits, assuming I can determine which. The other aligns it's 16bit funny in the frame and the DAC won't accept it, need to look into that one.I might have to just go to 24bit on a 32bit frame and 96K just to keep a flat format in, through and out. For the consumer analogue end I really don't need more than 16bit/48K but for the mixing and filtering it would be very advantageous to use bigger more precise floats/doubles and 2 to 4 times Nyquist and round it off at the end.I mean the I2S abstraction I wrote could handle multiple stream formats and config, each buffer would be labelled with it's format so something sensible happens in the right places to it. I can go there IF I have to, when I have to.

author: Siwastaja
author_rank: Super Contributor
timestamp: January 21, 2023, 12:33:06 pm
content: With a microcontroller (i.e., not writing mixing software for general purpose PC), you have the option to filter / mix in periodic ISR, reading the single values from input buffers, mixing (calculate one output sample), then outputting the value to I2S FIFO directly. So basically no DMA at all. The advantage is simplicity of code: no management of blocks / buffers, and of course, minimized latency from input to output (if you do mixing of realtime signals).In general purpose computing, even in kernel space but especially in userspace, you have the risk of mysterious delays of many microseconds, even milliseconds. That is why audio is processed in block buffers. With standalone DSPs or microcontrollers, you have the luxury of direct processing of data.

author: paulca
author_rank: Super Contributor
timestamp: January 21, 2023, 02:13:56 pm
content: So this diagram is exagerated, however I think it shows no matter how long Time t is asynchronous streams will not remain aligned such that you can mix them within Time t (red arrows are valid mix window beginnings).. I don't think it maters if they are 8 bytes or 192bytes or a mega byte.Also while this prototype has 2, maybe 3 I2S inputs, ideally I would like more in the final project with more complex routing and multiple processing buses. I can certainly mix it all up in one big soupy code base or I can componentize it, async it, block buffer it and abstract the complex stuff out of the way of the important stuff and ideally create a modularised design which can be "plug and pray", update the config extensible.I realise these concept are some what alien in the MCU/Realtime world. As you said, excessive power overheads. I can waste lots of CPU and memory with layers of abstractionto make my life easier and my code stronger.Anyway, the answer to my original question looks like a confident, yes, as long as I don't dilly-dally in the ISRs I should easily be able to carefully abstractly flip a pointer

author: Siwastaja
author_rank: Super Contributor
timestamp: January 21, 2023, 02:29:10 pm
content: Quote from: paulca on January 21, 2023, 02:13:56 pmThis is what business management believes is true, but it really is not; this mindset has ruined computing. Instead, make your life easier and code stronger by doing things as simple as possible, so that you understand what is going on. This is definitely possible with audio DSP stuff, it's not rocket science.Mixing asynchronous signals will require some kind of real-time resampling algorithm. Simplest, nearest neighbor, can be trivially implemented in said ISR based approach: keep copy of latest sample for each input stream. If new sample is not available, then use the previous one.

author: NorthGuy
author_rank: Super Contributor
timestamp: January 21, 2023, 03:29:41 pm
content: Quote from: paulca on January 21, 2023, 02:13:56 pmObviously, if your receiver is a slave and the streams have different clocks, they will get out of sync - a number of samples transmitted per minute will be different. But they have good crystals, so the difference will not be dramatic. You can determine how big the difference is. Say, if the max difference is 20 samples per minute and your mixer wants to work for an hour, the difference will be 1200 samples at most. You would need a buffer longer than this. And you need to fill in the buffers, so that each contains 1200 samples before you begin producing the output.By the end of the hour, the slowest transmitter will be 1200 samples behind the fastest one. This is just because they clocks were not ideal. You shouldn't follow their clock errors. You just mix them sample by sample, and output the result with your own "ideal" clock.

author: paulca
author_rank: Super Contributor
timestamp: January 21, 2023, 03:52:44 pm
content: We can debate that all day, but mark my words, those days are numbered. The problem is the markets are changing. It used to be the case that, given hardware is pretty "fixed" in nature so can the code that runs upon it be, written once off bespoke and never changed for the life of the product! I mean that is one of the main advantages, stability and robustness through having read-only code and no general purpose read/write OS to mess things up or maintain. It used to be that the MCUs an embedded code where doing relatively simple things down in the bit bashing with rather simplistic <1000 line project with a single author.However those days are gone. The market now demands not only that your device work and play nicely with others... which are likely to diversify and be subject to change, but also the market now demands things like CICD and weekly production releases straight out to the 10s of thousands of end users instantly as a firmware update with a simultaneous roll out on the clouds/server side. The market is demanding such rapid pace of development that the only way to achieve it is larger and larger teams, more and more code reuse and frameworking (aka here's one I prepared earlier). When no one person has a total line for line understanding of the software the 1980s approach fails. Not instantly, but in a slow painful death as team members leave the company to avoid working on it anymore.Anyway. Here, by example is just how complex and wasteful the buffer pool is. It's 40 lines of code! Now I no longer need to worry about where buffers come from. If I have a problem with the buffers I know where to look/fix. I don't need to go fixing the buffer handling code across the whole set of ISRs at once.EDIT I'm not sold on the zeroBuffer thing yet. I think it creates as many corner cases as it addresses when compared with using NULL pointers for unavailable buffers.

author: DavidAlfa
author_rank: Super Contributor
timestamp: January 21, 2023, 03:57:14 pm
content: To me there isn't any difference between circular double-buffering and circular half-buffering, only "Two buffers that can be in different ram areas" and "two buffers placed consecutivelly".The DMA ISR will generate a interrupt when it finishes the current buffer and starts the second one, it's same ping-pong game!If the ISR are too fast, the fix is obvious, increase your buffer size at the expense of a slightly higher output delay.

author: paulca
author_rank: Super Contributor
timestamp: January 21, 2023, 04:26:02 pm
content: I realise that MCU people conflate FIFOs, queues, circular buffer and double buffering to be the same thing. They aren't. Related, often implemented with, using or as one-another but not the same.Circular buffer != double buffering.A FIFO ~!= a circular buffer. A FIFO maybe implemented using a circular buffer, but it can be implemented other ways. The same for a LIFO or a stack.A Queue != FIFO and so on....It matters not where the memory is. They are implemented differently, have different pros, cons, synchronising and concurrency considerations. Most are in some ways defined as software engineering design patterns with distinct properties.No matter how many times an MCU datasheets says there are the same thing is doesn't make it so.

author: paulca
author_rank: Super Contributor
timestamp: January 21, 2023, 04:40:27 pm
content: Quote from: DavidAlfa on January 21, 2023, 03:57:14 pmI'm not ping-ponging. I have more than 2 time slots in play. It's not just IN buffer OUT buffer. There is the alignment/mix/eq buffer.The only two solutions are:* memcopy* multi-bufferingThe mechanics are so close I realise it's confusing. In the case of a DMA circular array the array is fixed in memory. Anything which is to go into it or out of it must be moved to that fixed place in memory. The entire copy process has to proceed uninterrupted or the dragons will be loose. Any mistakes, hang ups, missed interrupts and I get blasted with noise.In the case of multibuffering, there can be as many buffers as I want, they can exist in many different arrays of memory, they can concurrently be in use across many different "threads" without any of them ever being in the same bit of memory or in any contention. The DMA stream is not "fed" data into it's plate while it's eating from it, it's pointed to a whole new plate while the last one can sit on the counter until the dishwasher is ready (tangent analogy!)It also provides aggregation decoupling if I so needed. If I had more than one core that could be fun. I could collect a bunch of buffer pointers into a batch and gain processing efficiencies and the beauty is, the client code doesn't need to know a single thing about it.

author: NorthGuy
author_rank: Super Contributor
timestamp: January 21, 2023, 05:02:18 pm
content: Quote from: paulca on January 21, 2023, 03:52:44 pmBuffer pool doesn't require random access and often is served better with a linked list, for example:Code:Buffer_t* getBuffer() {Buffer_t* temp = freePool;if (temp) freePool = freePool->Next;return temp;}void freeBuffer(Buffer_t* buf) {buf->Next = freePool;freePool = buf;}This is less code, easier to write, works faster.Now tell me. Why the market demands bloat, such as zeroPool, instead?

author: paulca
author_rank: Super Contributor
timestamp: January 21, 2023, 05:12:24 pm
content: I'm not throwing away your idea Siwastaja.Thinking through the implementation it's effectively a single ISR. I2STxFifoEmpty or whatever.A for loop round the input I2S to lift their FIFOs, mixing as I go with += and then the actual EQ computations is a few dozen FPU instructions, a few hundred clock cycles and we write that to the output I2S FIFO and go back to sleep... or computing the coefficients and outputing metrics VU meter and stats to the screen.The point on the synchronisation being inherent in it is a valid point. It automatically introduces padding or truncating. Late data, the previous sample gets reused (fine at the sample level, not at the block level). Early data overwrites the FIFO which is exactly the same as shortening the buffer by 1 frame to resync.The concerns I have are around the margins and scalability.Those I2S FIFOs are very short. With 32bit stereo frames consuming the entire FIFO for a single stereo sample works out about 10us per FIFO.So I have no doubt it's going to work for 2 streams, maybe 3, but when it gets to 4 input streams and 2 output streams with 2 different EQs and some routing. It could get very tight.So it's not that it's an bad approach its just that it's very brittle and when it's capacity is exceeded there is little to no way to extend on it other than a faster micro or slower I2S.

author: paulca
author_rank: Super Contributor
timestamp: January 21, 2023, 05:25:36 pm
content: Quote from: NorthGuy on January 21, 2023, 05:02:18 pmBecause your version is unbounded, unsafe, leaking and a NPE exception waiting to happen. Nor is it sequentially thread safe. Your buffers will get allocated and freed out of order and your linked list will corrupt. When you wrap it up with error handlers and corner case conditions it will be a lot longer that the fixed array version.Mine uses a finite set of buffers which are deliberately IDd such that they can be individually accounted for. It also makes several of the buffer operations atomic. Out-of-ordering does not matter. Buffers are not allocated and discarded but reused. There is no dynamic allocation (except in init) required.The zeroBuffer isn't bloat, it's a fail safe. The idea is that the pointers are NEVER NULL. It removes the entire ability for a NULL pointer issue by removing NULL and replacing it with (I would hope) a constant buffer of zeros. Putting that into the bufferpool however introduces other issues like handing the zeroBuffer to an Rx peripheral for example and other issues, so I'm dropping it.Seriously. How many threads in this forum are a wall of tears from MCU people writing 1980s code and then getting all f'ked up by concurrency issues? Hmm? All of these things are simples until suddenly they aren't. Then tears. Why are you attacking me for doing it properly in the first place and engineering it such that I do not encounter the same tears. Designing it in from the start. As my day job is on big multicore distributed iron it's my trade to work these things out in ways that don't result in tears later. You would be surprised how rare that is and how often it's done piecemeal and a liability.

author: NorthGuy
author_rank: Super Contributor
timestamp: January 21, 2023, 06:11:52 pm
content: Quote from: paulca on January 21, 2023, 05:25:36 pmIt's safe unless you deliberately try to break it.Quote from: paulca on January 21, 2023, 05:25:36 pmnot leaking. Show me a code where it would leak.Quote from: paulca on January 21, 2023, 05:25:36 pmBut that's what you want. If you try to allocate a buffer, get NULL, but try to use anyway, the program crashes making you aware of the problem so that you can fix it before it causes problems in production.Quote from: paulca on January 21, 2023, 05:25:36 pmNot really. show me a code which would do this.Quote from: paulca on January 21, 2023, 05:25:36 pmI won't.Quote from: paulca on January 21, 2023, 05:25:36 pmThere's no corner conditions.Quote from: paulca on January 21, 2023, 05:25:36 pmmine tooQuote from: paulca on January 21, 2023, 05:25:36 pmAlbeit useless, you can add Id to my buffers too.Quote from: paulca on January 21, 2023, 05:25:36 pmzeroBuffer is not thread safe. Two concurrent threads may get the same buffer. Need to disable interrupts to use in different threads. Same as with mine.Quote from: paulca on January 21, 2023, 05:25:36 pmSame with mine. THe pool is created once and buffers are reused. That's what the buffer pool does.Quote from: paulca on January 21, 2023, 05:25:36 pmVery bad idea. Accessing non-existing buffer is a bug. You don't want to mask bugs. Otherwise they may go unnoticed and will cause problems later.Quote from: paulca on January 21, 2023, 05:25:36 pmI am not attacking you. Somebody taught you very bad programming principles which are dominant today and produce bloaty and buggy software, but do indeed save money by letting companies to employ incompetent coders. I try to demonstrate that other ways exist. I don't expect to succeed really. Just hope that may be not all is lost and you can embrace sound engineering principles some day in the distant future.

author: paulca
author_rank: Super Contributor
timestamp: January 21, 2023, 07:09:12 pm
content: You know what, put in practical terms, it doesn't actually bother me.I mean we are getting close to cleaning up the last mess your style of programming left behind in the enterprises of the 1980s. The embedded and cloud spaces are converging and I don't think your coding style will win. I don't see enterprises pushing functionality down into the MCUs as they currently are. They simply won't accept that pioneering freelander style of coding. It does not scale. It doesn't scale in code and it doesn't scale in teams. We who have ventured forth into the world of millions and millions of lines of code and teams of hundreds or thousands have found... it gets exponentially harder as it scales. We have learnt the hard way that "hacked out code" does not scale. When requirements change, you need to rewrite it again and again and again. At some point someone is going to say, "STOP!", just take the god damn pre-canned, reusable, slightly slower dependency and stop reinventing the wheel! That leads you actual software engineering. A discipline which is far younger than the MCU is and exists exactly because there are very real and very hard limitations to hacking out low level code.I mean, lets not argue whether a linked list of array is approapriate. Technically, collections theory would suggest a set is the ideal container. However the easiest way to implement a set in C is via an array with integers as keys pointing to a fixed array. Which leads to the second qualifer for a linked list. That it allows items to be added, removed and reordered. None of which are required. It doesn't really matter that the array is ordered, it's really being used as a set.I can sort of see a differential in approach forming here. I design, I use patterns, I reuse patterns, I aim to create reusable patterns and then I consider how I implement that.I can see how watching that (this) process can be very frustrating for a monoglot embedded C engineer.Technically I am a guest in your domain in these forums, so I shouldn't be too critical.I am honestly in part just bantering with you. Nor am I saying my approach is the right one, just that I also don't think yours is either. I've got a few take aways from the thread. I've learnt few things, all is good.

author: Siwastaja
author_rank: Super Contributor
timestamp: January 21, 2023, 07:32:13 pm
content: Quote from: paulca on January 21, 2023, 04:40:27 pmThat's not the idea of DMA! The whole point of DMA is you can change the pointer and write/read to/from anywhere. (This seems to be something DavidAlfa didn't get, either; if you pre-configure the DMA memory addresses then indeed, there is not much difference between one buffer or double buffering mode. But double buffer mode allows you to change the other pointer while one part is accessed.)

author: paulca
author_rank: Super Contributor
timestamp: January 21, 2023, 07:33:59 pm
content: I mean, it all sounds familar to me the way these discussions always seem to go.Peter-H mentioned it also, if you give people too much information, enough to confuse and distract them, then the thread derails. If you give the too little information they ask stupid questions and propose stupid solutions.I recall having a horrible time explaining my heating system to a bunch of HomeAssistant fan boys. They repeatedly under estimated the overall picture, features, functionality and protections.I mean take any one tiny code device, someone picks up on it and says it's not right or isn't needed. You start to explain that exactly how, why and what that component is or does involved a dozen hours of design, dry running, redesigning, diagramming, documenting and doodling to finally choose that approach as either the best or in many cases the least sucky. However without knowing ANY of those particulars or even understand what the thing true does end to end, the average forum guy can tell you the choice was wrong.It's like telling someone you took the car to the shops and as their only experience in life is living in a city centre they might bork at you and say, "You don't need a car to go to the shops! LOL Garf, garf". However the fact you live in rural utah and the nearest shop is 50 miles away doesn't occur to them at all.Thankfully. In the day job surrounded by mostly other peer engineers I don't have as much difficulty. On my engineering approach and it being "one big conspiracy of management", what bollox are you talking? Why would management want bloated software? Management want working software, on time, on budget with value add too boot. If you keep giving them estimates to rewrite everything because you refuse to generalise and componentise then... you will be got rid of fairly quickly.The embedded / hardware / firmware space is a bit more screwed though, because your dependencies cost money per item and you sell products. So it matters a lot to your bean counters if you use a $1 MCU or an $8 MCU. Those are very, very hard limits if the product has a target retail price of only £19.99. The cost of 100,000 MCUs of up to $800,000 versus $100,000 actually will pay a lot of salaries. In my domain that is flipped entirely. The cost of engineers far, far exceeds the cost of the hardware. So engineers who can develop fast, reliable, adaptable code win, regardless of performance or install size. As long as it's good enough and meets the requirements.

author: paulca
author_rank: Super Contributor
timestamp: January 21, 2023, 07:37:28 pm
content: Quote from: Siwastaja on January 21, 2023, 07:32:13 pmAgreed. What I'm trying to get at is circular mode is literally just a feature that STM32 added, it has no relation to DMA at all. The fact that STM32 choose to offer a circular mode using an array and 2 halves is an implementational detail decoupled for the concept of DMA. It would be equally valid to not provide a circular array "mode".Functionally speaking:HAL_DMAEx_StartMultibuffer( buffer, buffer )andHAL_DMA_Start( buffer );are different.

author: Siwastaja
author_rank: Super Contributor
timestamp: January 21, 2023, 07:41:29 pm
content: Quote from: paulca on January 21, 2023, 05:12:24 pmBetween batch processing of buffers, and processing one sample per ISR, there is nofundamentaldifference in scalability; they scale the same, namely: linearly. With one sample per ISR, there is ISR entry/exit overhead, plus some SIMD optimization opportunities go out of window. But it's not amassiveorders-of-magnitude difference, and the gap does not widen if you increase the number of streams, both solutions scale linearly.Having any FIFO at all helps. We do these things even with more ancient peripherals which only have a single holding register.Now, the reason why you see batch processing options of longer buffers being used, is that youneeda dedicated DSP/MCU to do per-sample interrupt thing. This is not always an option.But you have chosen an MCU, so there is nothing wrong doing it MCU way. Quite the opposite, the timing is IMHO easier to analyze, as you have already seen. You have 10us. Measure how long it takes to do the processing. Add some safety margin. Now you know what happens if you double the number of channels. Not exactly, but pretty close.

author: voltsandjolts
author_rank: Unknown
timestamp: January 21, 2023, 09:15:14 pm
content: Quote from: paulca on January 21, 2023, 07:37:28 pmEh? Continuous circular mode DMA is ideal for peripheral data reception into a large circular buffer, works well in STM32. Background code polls the ring buffer to see if any data has arrived and processes it as needed. Sometimes interrupt reception is a better fit for the application though, "it depends".

author: NorthGuy
author_rank: Super Contributor
timestamp: January 22, 2023, 03:12:32 am
content: Quote from: paulca on January 21, 2023, 07:09:12 pmI don't have a coding style. I simply write code which is required by the design.Nor do I think that there's a war of coding styles where something is winning or losing.Quote from: paulca on January 21, 2023, 07:09:12 pmI've been hearing this since at least early 90-es, so 30 years give or take. Complete non-sense. Wheel is already invented and you cannot re-invent it. But you can make wheels better. Re-using tricycle wheels on everything from trucks to Ferrari is not such a great idea.

author: paulca
author_rank: Super Contributor
timestamp: January 22, 2023, 08:31:45 pm
content: Anyway, that didn't last long.The I2S was receiving data yesterday. Today. Nought. It's clocking, it's not timing out, but all it gives me is 0.I have the full SPI trace on teh scope and it can decode it. It's the right format. I've tried both I2S1 and I2S2 and both produce 0s. It was working yesterday.I've gone through all manor of combinations, speeds, clock settings, masterclock, no masterclock, I even dropped the CPU speed down to 120Mhz. Nope. Audio rate down to 48K and .. nope.Putting it down tonight.Two ways to confirm things:1. I'm going to connect an F411 back up to the same setup tomorrow. If it works straight away it's likely I zapped the STM32H7. I've only had it a few weeks and used it twice.2. I'm going to connect another I2S or a basic SPI source to it and see if that works. I had it receiving from this setup before, but you never know.

author: DavidAlfa
author_rank: Super Contributor
timestamp: January 22, 2023, 09:37:55 pm
content: So you're coding on the fly and now you can't point the culprit of the failure?Make a local git repo for any serious project, you'll never miss those "WTF did I change?" thoughs at all.

author: Siwastaja
author_rank: Super Contributor
timestamp: January 23, 2023, 06:45:51 am
content: This, use version control.And whenever you forget to commit any changes and struggle to find what broke working code, don't admit it online!

author: paulca
author_rank: Super Contributor
timestamp: January 23, 2023, 09:31:44 am
content: I started a new project. Setup CubeMX with an I2S.... 0s.I went back to the previous project that was working, 0s.So, no I haven't coded myself into a corner as I dont have any code in this project yet. Just one single buffer declaration and an HAL_I2S_Recieve()Other things may have changed, like I was playing with the formats on the optical side too. Although I pretty much went through all possible combinations, even though I can see the format on the scope.If it's zapped it's my own fault. I don't treat them with that much care. I wire them up while they are running and they tend to sit around on the desk wired up with the cat standing on them and things. I might need to show a bit more care.

author: Siwastaja
author_rank: Super Contributor
timestamp: January 23, 2023, 09:40:55 am
content: I have seen an STM32 MCU physically fail in a very peculiar way with internal damage but working IO cell: namely, an IO pin worked perfectly in GPIO mode for both input and output, but once SPI peripheral was enabled as alternative function, only zero was read (or was it all ones, I forgot). Code was good and replacing the chip solved the problem.But such issues are rare. 99% of the time it's in code. Which was also why I struggled with that one so much.

author: paulca
author_rank: Super Contributor
timestamp: January 23, 2023, 10:06:34 am
content: Quote from: Siwastaja on January 23, 2023, 09:40:55 amIronically the post you made on here was the third google entry when I searched.I'm not 100% sure yet, but I'm running out of options. I'm 90% sure the software is not at fault as I've tried about 3 different projects that were receiving data and are now producing 0s.I just need to complete the other side of the possibilities tree by using a different I2S source. It could be an electrical issue a that I'm not seeing. There is a leaking SMPS feeding 100VAC@1M into the grounds. I did at the start of the setup yesterday connect the I2S lines to the MCU, while said SMPSU was in circuit, lines first. GND second. I actually forgot they were being powered by different power supplies and forgot I needed a GND.I know if everything is hooked up the amount of current flowing around from the SMPSU is literally microscopic and not a concern. Where my concern lies is in when I connecting things up. The MCU is earthed via the GND on the programming header. The rest of the system is floating with the SMPSU. Is it a likely failure mode if I was to, say, connect the MOSI line between these two circuits first. That MOSI line will momentarily appear to be at a much higher voltage than 3.3V. It won't carry much current sustained, but what capacitance is there and how much current flows through the data pin while it's equalizing. Again, I probably wouldn't be concerned if I'd connected the GND first as it's just a bit of copper to earth and will carry the microcamps with no care in the world. The data pin front ends on the MCU may not though.And... no the scope ground was no connected at that point either which would have at least been pulling it all down to earth.

author: DavidAlfa
author_rank: Super Contributor
timestamp: January 23, 2023, 04:02:39 pm
content: Tell me you're not using a breadboard full of wires!15 years ago I was burning a lot of GPIOs in my PICs due that.A wire slips, bang! Here goes 12V to the pin.I remember labeling my PICs, marking the bad pins in red or directly cutting them.They weren't cheap either, the 16F84 was ~$6 sourced locally, I wasn't working either so I had to improvise, soon after I discovered the free samples!BTW today was my first work day in 3 years, after a very long hell of recovery!

author: paulca
author_rank: Super Contributor
timestamp: January 23, 2023, 06:12:41 pm
content: The breadboard amounts to 2 modules, 3.3V power and I2S interconnects. No 12V or even 5VThe H7 SPI1 works, but not I2S. Nothing I do makes it do anything but read 0.I confirmed the stream with an old project and an F411 received the I2S and forwarded it to the H7 over SPI. That all still works.SAI1 Block A appears to work. It's reading something at least.To be honest, it's maybe fateful. I really should be using the SAI blocks anyway. It is they who have the FIFO threshold interrupts available.I've run out of things to try. I can strip comments from the code MX generates and post it later if it would help, but I'm off for dinner and some WoW.Attached is the state the I2S always ends up in. It says it's working to me. It's RxFIFO is not empty because it's stopped on a breakpoint, it's raised it's RXP and it's data register is a big fat 0x0. It always is. As to why it's also saying it has a Tx packet and a duplex packet I'm not sure.Note, no DMA my code is literally just:uint32_t buffer;HAL_I2S_Receive(&hi2s1, (uint16_t*) buffer, 96, HAL_MAX_DELAY);I get 96 0s at the start of the buffer.

author: paulca
author_rank: Super Contributor
timestamp: January 23, 2023, 10:06:22 pm
content: So, SAIs (code below) works fine.It's abit micky mouse, but it's Siwastaja's idea basically. Not even bothering with interrupts.And as mentioned, when you change the incoming sample frequency or even change it to 24 bit it just carries on. The exception being all my filters are tuned for 48k so sound sh1t.I only have 1 source, but I don't expect a few being a problem.As you can see I added a completely unnecessary bit of load in the first of 8 filter coefficient calculations. It won't do 16. It does 8. Plenty of overhead.The trickier part is you can't have all you can eat with 4 SAIs each with 2 blocks. There are restrictions I still learning what I can't and can't do.Code:while (1) {HAL_SAI_Receive(&hsai_BlockA1, (uint8_t*) buffer, 8, 10);/* Take the smug smile of the core's face */for(int i = 0; i<8; i++) {PeakFilter_SetParameters(&bassBoost, 50, 25, 1.6);}for (int i = 0; i < 8; i++) {int32_t sample = buffer; // FIX ME try floats/doubles, the maths might work better.sample = PeakFilter_Update(&bassBoost, sample);sample = PeakFilter_Update(&bassBoost1, sample);sample = PeakFilter_Update(&midCut, sample);sample = PeakFilter_Update(&highMidCut, sample);sample = PeakFilter_Update(&highBoost, sample);/* I am not a compressor and Im not even compressing till the compressor comes!*/while(sample > 0x8FFF ) {sample = sample/2;}buffer = sample; / precision loss.}HAL_SAI_Transmit(&hsai_BlockA2, (uint8_t*) buffer, 8, 10);

author: NorthGuy
author_rank: Super Contributor
timestamp: January 25, 2023, 04:14:47 am
content: Quote from: paulca on January 23, 2023, 06:12:41 pmThe OVR bit is set, which means it did receive enough of something to overrun the buffer. So, it sees a signal on the clock line. The fact that you receive all zeroes means that the data line is always low (as seen by the SPI/I2S module). Possibly a wrong pin or a bad contact.You can try to connect the data pin to VDD instead of the signal and see if your zeroes change to ones. This might give you an idea where the problem is.

author: STM32
author_rank: Newbie
timestamp: May 24, 2023, 01:00:09 am
content: Very interesting, informative topic.Quoting the OP here:QuoteI totally agree with you! That's why I do not turn to discussion forums most of the time and I often find myself short of a solution, until I figure it out myself. Of course, it would be great to get some help, but most of the time, you will just get into an endless debate, that more likely ends up like a competition, and not "discussion". Well, my experience with another forum went way beyond the "too much info" and "too little info", as stated by the OP. It was way more horrible than that!Well, in getting back to the topic, I believe my problem is very similar to the OP's. However, I'm not into mixing multiple I2S streams at the moment, it's just a simple audio playback. I just wanted to implement it using the double-buffering approach, without circular mode, halving a single buffer. Even on a low-end device, this approach seems more viable to me in the sense of having plenty of time to do processing, decoding, or whatever on bufferA while bufferB is being sent to the decoder by the DMA. I was struggling for the past couple days to figure it out, but to no avail.I'm using the BSP driver code, which is based on the DMA in circular mode. First of all, could someone explain to me what is the difference/purpose of 'BSP_AUDIO_OUT_ChangeBuffer()' when compared to the 'BSP_AUDIO_OUT_Play'? As far as I can understand, it's only the macroCode:DMA_MAX(Size / AUDIODATA_SIZE)used on the latter. It took me also another day to find out that one takes nSamples while the other takes nBytes. This absolutely matters when you feed it with stereo samples in an int16_t ring buffer.Aside from that, I still couldn't figure out what might be wrong with my setup. I was probably one of those confusing circular-mode DMA with double-buffering. I'm going to try it with different DMA configuration and see if that solves the audio playback issue when using double-buffering (two separate buffers, not halving a single one).By the way, I might be interested to learn more about the feasibility of your multi-channel input/output/mixing/EQ approach on a single STM32. I believe you're working on a device with a large package, QFPA176, or BGA, to get 4 I2S streams available. On most smaller packages, I2S signals are muxed with many peripherals. In my situation, only one I2S is available. Most peripherals are used in the project, SDIO, QSPI, SRAM, LCD, 3 UARTS.

author: peter-h
author_rank: Super Contributor
timestamp: May 24, 2023, 10:33:08 am
content: QuoteHow is that even possible?Could the CPU contain undocumented FLASH bits which enable some of the peripherals (so STM can sell the same silicon at different price points and with different P/Ns) and these got programmed with a runaway program?As a general comment in the circular mode, it is very useful for waveform generation, going timer -> DMA -> DAC. Also useful for receiving data but more complicated (than a straight rx isr) to handle an rx overrun.

author: Siwastaja
author_rank: Super Contributor
timestamp: May 24, 2023, 11:13:16 am
content: Quote from: peter-h on May 24, 2023, 10:33:08 amBeats me. ESD damage is one possibility, but it would have to damage some internal structures (input stage transistors at SPI peripheral, or at the MUX leading to the SPI peripheral) without blowing up transistors responsible of the GPIO functionality. It's not that far-fetched - basically the pin is fanned out to many paths leading to different internal parts, and unless the ESD even is strong enough to killallof them, it could kill just the weakest one (some random path, depending on random process variation).QuoteNot impossible, but pretty unlikely, the firmware did have flash programming; runaway flash programming would have likely destroyed something else than just this.Although it was such freak thing that even most unlikely scenarios are possible.

author: paulca
author_rank: Super Contributor
timestamp: May 24, 2023, 11:32:21 am
content: Some MCUs use "fuses" for configuration. Is there a possibility and over-voltage spike on an IO pin "burnt a fuse" into a different config?

author: Siwastaja
author_rank: Super Contributor
timestamp: May 24, 2023, 11:38:25 am
content: Quote from: paulca on May 24, 2023, 11:32:21 amSome kind of undocumented actual fuse or flash-based "fuse" as suggested by peter-h, yes, possibly. As far as I know, there is no user accessible fuses that would cause what I saw.

author: Georgy.Moshkin
author_rank: Frequent Contributor
timestamp: May 24, 2023, 02:32:28 pm
content: Quote from: paulca on January 17, 2023, 01:45:40 pmYou just need the right information on how to use circular arrays optimal way. Over the past few years I used circular buffers for adc/dma/i2c/i2s/spi/uart/timer/dcmi. And not just for hobby, spi was for ethernet chip, and there was a loopback test. For uart it was dma bridge with protocol conversion, and it must run for days without glitches, for i2s it was audio over radio, and there should be time left for simple haar compression and resampling at receiving end, because either transmitter or receiver mcu had a little higher crystal frequency.Here is the secret: you copy data to safe buffers on "half complete" and "complete interrupts". Processing in interrupt is an exceptional case. Safe buffer is then processed in the main loop. Safe buffer size is arbitrary. For low frequencies such as i2s you can even use memmove to keep arbitrary buffer of arbitrary size updated in a way of "sliding window". For faster signals you can use two indexes showing from where to where data need to be processed in a safe buffer, which is essentially is a large software circular buffer.You can think of circular buffer as an external mcu which sends you chunks of data at some frame rate. Size of chunks does not matter, it's the same as copying files by chunks, file size may be arbitrary. If you are sending data over I2S it is the same, you prepare data in main loop and circular DMA will eat it by small chunks, you just need to put the data in right place at the right time. If "half complete", you know that the best time to fill first half with new data is came. When "complete" interrupt fired - the best time to fill second half of DMA buffer is started. Moreover, you can use MEM2MEM dma transfers to free up all this time in main loop for data processing. Or you can use memmove/memcpy, but these are too slow if you are pumping data from dcmi to spi, or just reading adc using dcmi or PSSI.I did a lot of such data pumping and reliability tests, e.g. output safe buffer indexes through uart and showing percentage of unprocessed data in larger "safe buffer". Everytime the same approach.Your approach may be interesting, but I am not convinced. Your list of possible problems reminded my own rants of younger age, I just needed the right information on many topics, but there was no ready made solutions, very frustrating. About hear damaging I would argue that rewriting DMA registers to make your approach work have more risk of garbling things.Below is some code and links that may be useful.Code:#define ADC_BUFFER_SIZE 1024// do not forget to place these buffers in DMA-accesible RAMuint16_t adcBuffer ; // for ADC circular bufferuint16_t adcBuffer2; // copied from adcBuffer using mem2memvolatile uint8_t idx=255;void m2mCallback(DMA_HandleTypeDef *_hdma){// you can track mem2mem complete here, something likeif (idx==2) {idx=3;}}// ...HAL_DMA_RegisterCallback(&hdma_memtomem_dma2_stream0, HAL_DMA_XFER_CPLT_CB_ID, m2mCallback);while (1){idx=0; // allow mem2memwhile(idx!=3){asm("nop");// process data here// if processing fast enough, idx will be set to 0 before// HAL_ADC_ConvHalfCpltCallback// will be called again// idx=0 - need to copy, 1 - first half copy initiated,// 2 - second half copy initiated,// 3 - second half copy mem2mem completed}void HAL_ADC_ConvHalfCpltCallback(ADC_HandleTypeDef* hadc){if (idx==0) {// memmove( (void *)&adcBuffer2,// (void *)&adcBuffer,// sizeof(adcBuffer)/2);HAL_DMA_Start_IT(&hdma_memtomem_dma2_stream0,(uint32_t)&adcBuffer,(uint32_t)&adcBuffer2,sizeof(adcBuffer)/2/4);idx=1;}}void HAL_ADC_ConvCpltCallback(ADC_HandleTypeDef* hadc){if (idx==1) {//memmove( (void *)&adcBuffer2[ADC_BUFFER_SIZE/2],// (void *)&adcBuffer[ADC_BUFFER_SIZE/2],// sizeof(adcBuffer)/2);HAL_DMA_Start_IT(&hdma_memtomem_dma2_stream0,(uint32_t)&adcBuffer[ADC_BUFFER_SIZE/2],(uint32_t)&adcBuffer2[ADC_BUFFER_SIZE/2],sizeof(adcBuffer)/2/4);idx=2;}}https://community.st.com/s/question/0D53W00001hjcYgSAI/stm32h747-dma-linked-list-or-double-buffering-and-adc-interleavedhttps://community.st.com/s/question/0D53W000016nboxSAA/spireceive-to-usbtransmit-via-dma-on-stm32h7

author: paulca
author_rank: Super Contributor
timestamp: May 26, 2023, 08:25:24 am
content: In my trade we pick patterns based on the problem being solved.CircularBuffering or RingBuffering are one type of buffering pattern.Double buffering is a different pattern.Triple Buffering is a similar pattern.n Buffering isn't really a thing. Because that is the combination of two patterns. A collection of buffers usually has an ordering mechanic to it, thus it is a Queue. A Queue is a high level pattern type, like Buffer and could be a FIFO, a LIFO or a dynamic queue. A fairness queue, a priority queue, a transactional serialising queue, and inoperatability queue....Alternatively you could use a "pool" pattern around your buffers. Such that you have have pools of free buffers, busy buffers and ready buffers. A pool is basically a "bag" collection. You just want one of the right type, so you reach into the bag and take anyone. Yes that is an acedemic definition of an unordered collection where access to individual items is unavailable and only a random item may be recovered. Similar to it's more specialised cousin, the Set which is a bag, which cannot contain duplicates, but you can only ask for a specific item, or take one out at a time.The HAL Libraries come with a Circular pattern which is so broad grained it looks almost like a double buffer as it only have 2 "sections". The difference between this fairly rudimentary "circular" buffer and "double buffer"'ing is semantics and details, but depending on where and on what you are writing they can become extremely important.They also come with the DMA "Double buffer" mode. And as you can call it sequencially with different buffers that allows for you to implement double, tripple buffering. If you need more buffers, then you need to think in terms of queues or pools.You may all be right in that none of the above engineering is required. As it turned out this whole implementation, thanks to advice on here, went a completely different direction entirely. With no buffers. Processing sample by sample in real time in the interrupts!EDIT:Also of note. Technically a circular buffer, if done correctly will allow WAY more than a fixed 2 independant access points. In fact a very, very basic low level pattern is to have every element in the concurrent array read/write individually. So your 256 byte array can have a read and write pointer for each element, protected for each other. You can ever have multi-readers and multi-writers.The parent topic to the above or where it is usually laid out, is in the Producer, Consumer patterns. Slow consumer, fast producer. Many consumers, single producer... and so on and so forth. With in that you find all the different buffering, pools, queues and other patterns emerging.