Direct digital synthesis for music? - Page 1
Forum section: electronics / projects

author: technix
author_rank: Super Contributor
timestamp: September 28, 2017, 05:22:59 pm
content: Given the computational power of modern microcontrollers, is it possible to implement direct digital synthesis of multi timbre music on them? If so, what kind of interface and memory bandwidth, as well as data throughput, am I looking at?For example, is it possible to implement a MIDI module with stereo 24-bit 96kHz sound quality, 32+ voice polyphony and a sound bank of at least 400 instruments (choosing up to 16 from the 400 instruments), using the STM32F756 microcontroller and a CS4334 I2S DAC? How much sample memory would I need (on microSD card, NAND Flash or Quad-SPI NOR Flash) and do I need external SDRAM for the throughput?Background: I have a musical keyboard with a busted internal sound chip but a working MIDI output. I am okay with hooking it to the computer using GarageBand as the synthesizer, but when family borrows it they need a synthesizer too in order to produce any sound out of it. I am not that much of a musician but some of my family is, who have long looked forward to borrow the keyboard, but got put off by the need of a computer to synthesize the sound.

author: ataradov
author_rank: Super Contributor
timestamp: September 28, 2017, 05:35:11 pm
content: If you are talking about simple ADSR envelope synthesis, then it is very easy, and Cortex-M7 should be able to handle it with some carefully crafted code.If you are talking about sampled instruments, then one of the first challenges here would be getting that bank of 400 instruments.In all of this, I don't see a need for 24-bit and 96 kHz DAC. This is a total overkill.

author: technix
author_rank: Super Contributor
timestamp: September 28, 2017, 05:49:40 pm
content: Quote from: ataradov on September 28, 2017, 05:35:11 pmADSR is needed but not enough. There still have to be a seed waveform to ADSR on.Quote from: ataradov on September 28, 2017, 05:35:11 pmI do wish I can use a mix of samples and ADSR (Kind of like the Roland MT-32 LA synthesis?)For now I would just borrow someone else’s samples like GarageBand or Microsoft GS Synth. If this becomes a commercially viable product I will have to contact instrument manufacturers and create new sample sets.Quote from: ataradov on September 28, 2017, 05:35:11 pmWill the internal DAC of the STM32F7 work then? Or is it still better to use an external I2S unit?

author: ataradov
author_rank: Super Contributor
timestamp: September 28, 2017, 06:01:52 pm
content: Quote from: technix on September 28, 2017, 05:49:40 pmYou can ADSR each partial of the note, so you would be able to reproduce reasonably sounding instruments. The issue here is getting that ADSR data. For some instruments it can be extracted automatically from the recorder sound banks, but not for 400.Quote from: technix on September 28, 2017, 05:49:40 pmIn principle, I see no problem doing this. Memory throughout should not be an issue here. CPU bandwidth will be a bottleneck, especially for 32 note polyphony. But if you assume that most of the partials decay really fast and can be eliminated from the calculation fast, then it will get easier. This way pressing all 32 keys at the same time will not work, but normal performance will be just fine.Quote from: technix on September 28, 2017, 05:49:40 pmBut they are full samples. You will still have to derive data for sustained part somehow. That's either a lot of work, or some clever automation.In any case, the first 90% of the work will not involve microcontrollers. Make the thing work on PC first, and you will get an idea of resources required, plus you will have all the data required for synthesis.Quote from: ataradov on September 28, 2017, 05:35:11 pmI2S is easier for DMA, so I would still use it. Juts in 48 kHz 16-bit mode.

author: technix
author_rank: Super Contributor
timestamp: September 28, 2017, 06:27:09 pm
content: Quote from: ataradov on September 28, 2017, 06:01:52 pmCan I run FFT on recordings and get the parameters automatically.Quote from: ataradov on September 28, 2017, 06:01:52 pmHow much memory would it need then? Do I need like 64MB or 128MB SDRAM to hold the data? (Loaded from a microSD card.)Quote from: ataradov on September 28, 2017, 06:01:52 pmFor what I know the GarageBand samples comes with the ADSR part sof the sample already cut and the sustain part being suitable to loop. Will this help?Quote from: ataradov on September 28, 2017, 06:01:52 pmHow can DMA help for the polyphonic synth?

author: ataradov
author_rank: Super Contributor
timestamp: September 28, 2017, 06:34:03 pm
content: Quote from: technix on September 28, 2017, 06:27:09 pmI would go with QSPI and reading the data directly. I'd avoid SDRAM as much as possible.Quote from: technix on September 28, 2017, 06:27:09 pmIt will help a lot, of course, if you manage to actually extract this data. I've never looked at their file formats. Plus you automatically know thew size of the sound banks.But come to think about it, 400 instruments is probably a stretch for QSPI memory.Quote from: technix on September 28, 2017, 06:27:09 pmIt will help with getting the data out, not the synthesis directly.You will want to synthesize 128-1024 samples and let DMA take care of sending it out while you work on synthesis of the next block.

author: technix
author_rank: Super Contributor
timestamp: September 29, 2017, 12:35:38 am
content: Quote from: ataradov on September 28, 2017, 06:34:03 pmGarageBand samples takes gigabytes (~2GB for basic set, 10GB+ for the complete set.)

author: ataradov
author_rank: Super Contributor
timestamp: September 29, 2017, 12:43:49 am
content: Quote from: technix on September 29, 2017, 12:35:38 amThey are probably of very high quality, like multiple layers (velocities) and all that stuff. You may not need all of this. After all, I'm assuming you are not looking to design the most brilliant instrument in a world? Because in that case you are better off with a $300 x86 laptop, not Cortex-M7 MCU.You can do quite a bit with CM7, but full GarageBand sound would be pretty much impossible.

author: technix
author_rank: Super Contributor
timestamp: September 29, 2017, 01:04:52 am
content: Quote from: ataradov on September 29, 2017, 12:43:49 amNo this is not intended to be the most brilliant instrument (for that I would use at least a $2k MacBook Pro.) I mostly needs to make my keyboard make some noise when the keys are pressed.GarageBand samples are stored in RIFF sowt format with multiple layers and 24b@192kHz sampling. If I want to use it it needs downscaling anyway.For all I know you cannot DMA from microSD to I2S directly. This is where the SDRAM comes into play as I can just load the current instrument samples into SDRAM from microSD when loading.

author: calexanian
author_rank: Super Contributor
timestamp: September 29, 2017, 01:06:52 am
content: Look up a Fairlight CMI. it was an early computer built specifically to do just this. Interesting story. What people think is a Rhodes or other piano in Alan Parsons Eye In the Sky is one I believe.

author: ataradov
author_rank: Super Contributor
timestamp: September 29, 2017, 01:10:21 am
content: Quote from: technix on September 29, 2017, 01:04:52 amBut MCU will not support that much SDRAM anyway. You will be limited to 256 MB at best. So it may be possible, but will require some shuffling of instruments ahead of time.But you can DMA from a memory-mapped QSPI flash though.The hardest part is actual synthesis, unless you just want it to be sample-based.

author: amyk
author_rank: Super Contributor
timestamp: September 29, 2017, 02:24:39 am
content: Look up howtracker musicworks.

author: technix
author_rank: Super Contributor
timestamp: September 29, 2017, 06:47:24 am
content: Quote from: ataradov on September 29, 2017, 01:10:21 amThe QSPI window is 256MB too - not any better than SDRAM. Also I don’t have to load all the instruments all the time too, just load the patches from microSD to SDRAM upon selection. It is much faster to access SDRAM than both QSPI and microSD.

author: hamster_nz
author_rank: Super Contributor
timestamp: September 29, 2017, 07:45:42 am
content: Why not do a rough back-of-the-envelope calculation of how many multiply accumulate operations you need, and see if that gives any insights as to how much horsepower you need?For example, using the following assumptions:- upper limit of 64 concurrent sounds- playing back waveforms from memory for each note (so no filtering / mixing needed)- per-note volume (e.g. velocity)- 96kHz output sample rate,That needs a CPU that can do 6 million multiplications/accumulate per second, and 6M memory accesses per second (so 12MB/s). That is sort of fast micro-controller territory.Such a design would need a lot of memory - for about 200 different voices, 100 different notes per voice, and the average sample being 100k 16-bit samples, that is 4GB of memory (but reachable with a SD card). You can then trade off memory requirements off against MACs until you find what is a sweet spot for your design.

author: technix
author_rank: Super Contributor
timestamp: September 29, 2017, 07:46:16 am
content: Quote from: amyk on September 29, 2017, 02:24:39 amI think using GarageBand instrument patches would be like this, since the attack and release is one audio clip, sustain is a looping clip and release is yet another clip.

author: technix
author_rank: Super Contributor
timestamp: September 29, 2017, 08:12:44 am
content: Quote from: hamster_nz on September 29, 2017, 07:45:42 amFor me I can just keep the current patches in RAM and load them on the fly from microSD card. I am okay with some loading time before the music starts but stutter is not okay. The machine need to support up to 16 concurrent patches (the maximum allowed by MIDI anyway) and 32 voice polyphony.So processor speed is my least worrying at 12.288 MIPS using integer operations. Each patch needs four instructions. A 216MHz Cortex-M7F can deal with this.Memory speed is concerning. There are 32x 32-bit D-bus transactions and 1x 32-bit S-bus transaction each sample, so 12MB/s. I think this is doable with x16 PC133 SDRAM.Memory usage... Well... Each patch has 88x 2-second audio clips with 32-bit @96kHz (I need to align the bytes or access time would be atrocious and unpredictable.) So 135.168kB per patch and 2.163MB with all 16 loaded. This considers only a single velocity.If multiple velocity is added the amount of FMA operations and memory bandwidth doubles, and the storage requirement multiples.

author: JPortici
author_rank: Super Contributor
timestamp: September 29, 2017, 08:22:26 am
content: I'd rather get a raspi or simillar and launch the required software under linux. Did you see the mellotron on a raspi.I second every word in blueskull's post.

author: technix
author_rank: Super Contributor
timestamp: September 29, 2017, 08:28:53 am
content: Quote from: JPortici on September 29, 2017, 08:22:26 amThe relative of mine wants it to turn on and go. She is okay with a few seconds of loading time before starting a song, but a minute to load the OS is not quite okay.Also how do you get MIDI into the Pi without the notoriously unstable USB?

author: hamster_nz
author_rank: Super Contributor
timestamp: September 29, 2017, 08:35:06 am
content: I wrote a MOD file player for a PC a long time ago. The problem with the 'tracker' style of pitch-shifting samples (e.g. by dropping or repeating samples) to is that the quality of the output is really poor once you move too far away from the recorded speed.I suspect that the original Amiga audio hardware had a DAC for each voice, and DMAed the samples to them at different sample rates to change pitch.If the samples have high frequency content to make them sound 'nice' then dropping samples creates unintentional unmusical frequencies. You can improve things using digital filters to 'nicely' change the sample rates, but then that quickly becomes computationally expensive.If anybody knows a workable solution I'ld love to hear all about it!

author: borjam
author_rank: Unknown
timestamp: September 29, 2017, 08:39:46 am
content: Quote from: calexanian on September 29, 2017, 01:06:52 amWell, that solves the mysteryI was really puzzled by the Rhodes-but-not-quite-it sound in Eye in the Sky. It's a smoother, more polished version of the Rhodes soundwithout the typical harshness. I always wondered what it was actually.

author: technix
author_rank: Super Contributor
timestamp: September 29, 2017, 08:59:32 am
content: Quote from: hamster_nz on September 29, 2017, 08:35:06 amFor the samples I am using now there is one sample for each key - no shifting needed, but occupies a lot of memory.

author: borjam
author_rank: Unknown
timestamp: September 29, 2017, 09:55:49 am
content: Quote from: technix on September 29, 2017, 08:59:32 amProfessional sample libraries are usually huge. Remember that acoustic instruments don't have uniform characteristics for every note. Anddepending on the instrument there's a lot of variation. Even for a piano (which looks like a "simple keypress") there is a big differencedepending on the pressing velocity.For wind and strings it's even more dramatic. Listen to Kenny G (ugh!) and Andrzej Olejnikzac playing the soprano sax and you won't believe it's the same instrument

author: John B
author_rank: Frequent Contributor
timestamp: September 29, 2017, 10:09:10 am
content: What's the end goal of the project? There needs to be a reason to reinvent the wheel (curiosity and challenge is perfectly valid), however if it's to get a basic keyboard working again, perhaps a raspberry pi running Pd (pure data) or csound etc. It would be a sensible blend of challenge and functionality. Plus you could put your own colour screen in the keyboard. At the end of the day you're left with a keyboard with a basic action. When you get to the really high end, you buy a bespoke keyboard action separate from the sound generating bits....

author: Unknown
author_rank: Unknown
timestamp: September 29, 2017, 10:16:28 am
content: Quote from: calexanian on September 29, 2017, 01:06:52 amI remember seeing a demo by Bob Moog of the Fairlight, the highlight of which was playing a MIDI file using the 'Fart' sampleTo the OP, samplers up until recent times got round the memory requirements by using tricks likescaling a sample over a few keysusing a short sample and looping a section for sustainsample compressionusing core samples and filters for variationsusing layers to add realism like piano hammer soundsThe problem with using a PC is it MUST have a low latency sound card & driver. If there is any noticeable latency then it will be unplayable.Building your own sound module is pretty pointless unless you want to do it as an academic exercise, there are lots of used ones available for peanuts if you don't want the pro stuff.

author: simmconn
author_rank: Regular Contributor
timestamp: September 29, 2017, 06:56:58 pm
content: Someone in the retro PC game area has a solution that may work for OP. Search for Dreamblaster X2 (and Dreamblaster S2 for a standalone box).

author: Buriedcode
author_rank: Super Contributor
timestamp: September 29, 2017, 07:08:58 pm
content: I looked into this a while ago as I could never afford a decent synth and liked to compose with many different instruments. Except guitar because I have yet to hear a decent synth guitar.But as I was looking into a hardware solution, plugins, soundfonts, and 'free samples' got orders of magnitude better, leaving the only DIY thing to make for music, was MIDI controllers (which are still cheaper to buy than make, but its cool having sliders, knobs, pads, and proximity sensors).Playing sound from a micro, be it 8-bit of 32-bit is one thing, but managing multiple velocity samples, stereo imaging, and 64-note polyphony to make something approaching the quality of commercial products (hardware, or plugins) is a lot of work, for little gain, except perhaps interest. Although considered poor quality, I still like the Yamaha D50 and the Korg M1. Even doing those in hardware would be a task. Hardware aside, actually getting 'decent' samples used to be incredibly difficult without paying through the nose, but now with file sharing, and plenty of free soundbanks floating about...I did make my own sampler for a keyboard for my niece, so she could record sounds and play them back at different pitches. But that was a 'toy', and was done as a hobby as you can buy them for £10.tl;dr free plugins, soundfonts/samples played on linux/PC hardware will be easier, cheaper, and 'better' than DIY hardware solutions.

author: BrianHG
author_rank: Super Contributor
timestamp: September 30, 2017, 04:56:47 am
content: If I were to attempt this project, I would use one of these under 99$ SBC. (Single Board Computers) which come with everything you need assembled, including Linux...)http://linuxgizmos.com/99-sbc-runs-embedded-linux-on-1ghz-dual-core-arm-soc/Look through the list here:https://en.wikipedia.org/wiki/Comparison_of_single-board_computersYou should be able to find a dual core 1Ghz with a GB of ram, audio, serial & enough flash for your needs for under 60$us for 1 at a time. Less with quantity.Don't fool yourself, by the time you choose a smaller single core MCU, add ram, flash, audio DAC, SPDIF digital audio out as well, power supply circuitry, and everything else like PCB and assembly, you will be already in the 40$ region, not to mention working without an os. Note some 800Mhz A9 SBC start at around 39$ 1 at a time, but those might be too slow for high definition audio.

author: JPortici
author_rank: Super Contributor
timestamp: September 30, 2017, 05:51:05 am
content: Quote from: Buriedcode on September 29, 2017, 07:08:58 pmwho says it's poor qualityi love that old guy.fun fact, one of the lead programmers on that project (and many of the following korgs) is an italian guy, michele paciulli, he's one of the coolest dudes i've ever met, very funny and creative. sometimes you can see him at korg demo events around europeAFAIR the M1 used a 56k, they all used the DSPs available at the timeit did not..http://www.littlediode.com/datasheets/pdf/Datasheets-UPD/UPD70208H.PDF

author: helius
author_rank: Super Contributor
timestamp: September 30, 2017, 06:06:50 am
content: Quote from: borjam on September 29, 2017, 09:55:49 amA piano is the opposite of simple. Every string (each key is not just a single string) and the soundboard all resonate. The damper pedal is also quite unlike the hold pedal on a synthesizer. Something like a harpsichord or clavinet (or Rhodes) is much simpler acoustically.Every digital synthesizer since the DX7 has had "direct digital synthesis". Simply playing back samples is extremely limiting and usually not the approach used. Instead the source samples will be processed using resynthesis to extract sonic attributes, and stored in a wave rom. 16 bits for samples is total overkill, 8 to 12 is more than adequate since the waves will be shaped using ADSR. Waves are also commonly compressed or companded to reduce size and bandwidth requirements.

author: JanJansen
author_rank: Frequent Contributor
timestamp: September 30, 2017, 12:34:11 pm
content: Hi Technix, here you have a code for filtering :http://www.microchip.com/forums/m1003759.aspx

author: JanJansen
author_rank: Frequent Contributor
timestamp: September 30, 2017, 12:36:13 pm
content: Quote from: ataradov on September 28, 2017, 05:35:11 pmNo overkill if you want nice filtering, and no alliasing, especially the 96KHz.

author: technix
author_rank: Super Contributor
timestamp: September 30, 2017, 04:34:55 pm
content: Quote from: JanJansen on September 30, 2017, 12:36:13 pmWell it is necessary for me to use 32-bit samples anyway (as the ARM core loads aligned 32-bit data fastest, and it is more likely to get DMA to work on this size too) so 24-bit samples are no big deal anyway. The 96kHz may be optional though.

author: borjam
author_rank: Unknown
timestamp: September 30, 2017, 05:20:37 pm
content: Quote from: helius on September 30, 2017, 06:06:50 amI know! I said "looks likea simple keypress"Sorry if I wasn't clear enough. I wanted to say that it's not possible to just sample several notes and shift their frequency.

author: ataradov
author_rank: Super Contributor
timestamp: September 30, 2017, 06:12:05 pm
content: Quote from: technix on September 30, 2017, 04:34:55 pmARM core will load any data aligned o its natural size in the same time. So 16-bit data aligned at 16-bit boundary will be as fast.

author: borjam
author_rank: Unknown
timestamp: September 30, 2017, 06:27:34 pm
content: With a modern processor it's certainly possible to do very sophisticated synthesis. Just look at the modern emulations of analog synthesizers, for example. I own a real Korg MS-20 and the digital emulation, for example, and it gets really close.It differs when you try rather extreme adjustments that can cause non linear effects, of course. But in many cases the emulation is surprisingly faithful.Look at Arturia's Moog Modular emulation for example.

author: Jr460
author_rank: Regular Contributor
timestamp: September 30, 2017, 06:48:52 pm
content: I think you need to become friends with some of the engineers and sound designers at Kurzweil. Even once you have samples and the hardware, it takes a lot to turn that into a good sounding synth patch.For example, say you get a good sounding studio/room and good mics and record someone playing each key of a Yamaha C7 piano. At a minimum you need to normalize the levels of each sample. But then the timber changes depending on how hard the key is struck. You just can't use the MIDI velocity to scale your samples. Do you have someone hit it a 3, 4, 5 levels? When do you switch, do you interpolate between them. Also a piano key if held down, when do you call the timber consistent, and just loop that last part as the note tails out? Now you might have a huge set of samples. Maybe you want to pitch shift some. What is the max you want for that? You might be able get away with more in the high and low ends of the range, but need more around when the notes change from a single string to double and then to triple.Now take something like a Rhodes. How do to take care of the distortion from the pickups when you lay into it and make it knock? Listen to Steely Dan's Hey Nineteen to hear the many different characters of a Rhodes.I think one of the easier keyboards to mimic with DSS solution would be a Hammond B3. You are just summing up the waveforms for the the tone wheels scaled by the drawbars. Percussion settings are just adding a ADSR to some harmonics. Careful, the tone wheels in a real B3 are not exactly on the frequencies of integer multiples of the note. This is one thing that gives it that special sound. The other thing is a B3 without Leslie speaker (swirly box) is useless. So you need to take the raw B3 sounds figure out how to do the phase and doppler shifting just like the spinning speakers.The water gets deep real fast.Kurzweil, Yamaha, Korg, and Roland, Nord, all make very good instruments that you would be hard pressed to make on your own.

author: Jr460
author_rank: Regular Contributor
timestamp: September 30, 2017, 06:57:46 pm
content: Quote from: borjam on September 30, 2017, 06:27:34 pmAh the MS-20. I didn't own one, but used a friend's way back in the day. Thinking about it, that would be relatively easy to do, monophonic to start with. No samples, just a set of some simple or easy to create waveforms. a few ADSRs for things, and LFO. All the magic really happens in the filter. I understand there were two versions, one with Korg's chips that had a bit of hiss and distorted at times, and newer commercial ones that were cleaner sounding.

author: JanJansen
author_rank: Frequent Contributor
timestamp: October 01, 2017, 01:39:25 pm
content: With all the nice oscilloscope hardware you can read the EPROM of all those synthesizer wavetables.Maybe time for some nice audio subforum here ?