SD card emulator - Page 1
Forum section: electronics / general

author: encore2097
author_rank: Contributor
timestamp: October 01, 2016, 12:02:17 am
content: This might sound crazy but there is good reason for this and carefully considered for over a year from all angles, so if you're comment is "why not try X" without having understood the problem, please do not reply.With that said, I need to emulate an SD card via USB.The problem: creating an automated test and benchmark platform for embedded devices, specifically single board computers (SBCs).During development and testing we flash a bootloader + os image to an SD card on a host computer and then move the card manually to a target device (RPi or similar) and power it on to see if it works. If it works great, if not then we have to re-flash the SD card on the host computer and retry it. This is not feasible for an automated system and solution like SD-MUX (multiplexers) still require an SD card and can wear out and require switching which increases complexity.The proposed solution is an sd card emulator.It would have an SD/microSD form factor <-> micro controller + buffer <-> USB 3.0 <-> host computer with modified driver that uses a file as storage (emulated NAND)So how can we do this? The fastest, cheapest, most effective way to do this (theoretically) is to use a hacked USB 3.0 SD card reader.Let me explain, a USB 3.0 card reader uses a micro controller to read and write to an SD card. One example is the Realtek RTS5301 used on many USB 3.0 card readers and inexpensive.If the card reader using the SD pins can communicated with an SD card which stores and retrieves data on NAND storage, we should be able to mock out the NAND storage for storage over another protocol (in this case a filesystem accessed over USB).The card reader micro controller shows a SDXC/MMC 4-bit I/F on the block diagram and supported features include:• Support SD card clock 208MHz (max.) for SDR104 mode.• Support SD card clock 100MHz (max.) for SDR50 mode and 50MHz (max.) for DDR50 mode.If it is capable of R/W to a SD card at those rates, surely it must be able to emulate the opposite signaling to a host to mimic an SD card. Or at the very least drop down to a lower standard which is still faster than SD-MUX and more robust (wear and tear, switching USB, etc..)What do you think?

author: andersm
author_rank: Super Contributor
timestamp: October 01, 2016, 12:34:48 am
content: At work we're looking into using QEMU and QTest for running automated tests against generated SD card images. For running tests on the actual hardware, I would mount the filesystem over the network, if your OS and hardware allows it.

author: Rasz
author_rank: Super Contributor
timestamp: October 01, 2016, 05:03:14 am
content: or you could buy off the shelf $20 WiFi enabled SD card and upload files over the air.btw rasPee can boot from USB and ethernet now

author: abyrvalg
author_rank: Frequent Contributor
timestamp: October 01, 2016, 09:00:57 am
content: I'd follow Rasz's SD-WiFi advice, but if you still need to go long way:There are several FPGA-based SD emulator projects by the same guy on hackaday.io (https://hackaday.io/project/6946-sd-card-emulator-lattice-xp, check "similar projects" section also), but no sources published.RTS5301 will not work: it's a master device that clocks the interface, but you need a slave clocked by interface.Some full size SD cards have separate SD-NAND controller talking to big TSOP NAND chips (if you can emulate NAND bus. Cypress FX3?)There is a weird SD device with SPI flash instead of NAND (should be much easier to emulate - no scrambling, no ECC, no block mapping, I can see a plain FAT image in flash dump) - xtc2clip.org

author: mikeselectricstuff
author_rank: Super Contributor
timestamp: October 01, 2016, 09:11:19 am
content: Why do you have to reflash the SD card, as opposed to using another programmed one ?

author: encore2097
author_rank: Contributor
timestamp: October 02, 2016, 05:34:32 am
content: Quote from: Rasz on October 01, 2016, 05:03:14 amThis is an interesting idea.. I'm not aware that you can write to the SD card over Wi-Fi, but it should be possible since the card appears to be running embedded linux. However, there are no microSD cards with Wi-Fi capability. They do make an SD to microSD adapter/cable - if it works I have no idea.. I believe electrical pinout is the same..I'm also concerned with performance and the issue of wear and tear on NAND. Might be possible as a quick hack.Quote from: abyrvalg on October 01, 2016, 09:00:57 amI'm still new to EE so this might be a dumb question: whats unique to the slave device that the master cannot emulate? They communicate via the same electrical signals over the set of wires at the same clock speed. From my research, host initializes (powers) SD card and negotiates transfer speed, then it sends read and write commands which the SD card microprocessor interprets and answers by manipulating flash. Should be possible with a firmware hack to make the host act as a slave since it has all the capabilities to communicate with a slave or am I mistaken here?Re: NAND emulation, this would be very difficult and incredibly complex and I dont see the point to doing this.. NAND is used because of its form factor and cost, those are not constraints -- any data store will do. Besdies the host has no conception/idea of NAND it only sends read/write commands after initialization. So the emulation is the SD card interface and protocol. Taking the Wi-fi SD card idea its very similar to what I'm proposing.

author: encore2097
author_rank: Contributor
timestamp: October 02, 2016, 05:40:39 am
content: Quote from: andersm on October 01, 2016, 12:34:48 amWorks if you dont plan to modify bootloader / uboot / kernel. If you do plan to modify those ( say because your are adding in netboot or NFS) and it doesn't boot during development, get ready for a frustratingly long development cycle with lots of SD card swapping.Quote from: mikeselectricstuff on October 01, 2016, 09:11:19 amHope you are not trolling.. you are still swapping SD cards - which would not make it an automated test platform.The SD-MUX v2 uses this idea, it has two SD cards and two interfaces and electrically switches the connections between the target and the host. This still has the wear and tear issue and switching electrical connections has reported issues; intermittently not working one day and then working the next (maybe bug in HW or SW)

author: abyrvalg
author_rank: Frequent Contributor
timestamp: October 02, 2016, 07:13:52 am
content: Quote from: encore2097 on October 02, 2016, 05:34:32 amUnidirectional CLK signal (from master to slave, this is "hardwired" in silicon, like having a mouth to talk or having an ear to listen).

author: mikeselectricstuff
author_rank: Super Contributor
timestamp: October 02, 2016, 08:34:41 am
content: QuoteNot trolling - just trying to understand the actual problem, and why a big pile of ready-programmed and verified SD cards wouldn't solve the problem.

author: NottheDan
author_rank: Frequent Contributor
timestamp: October 02, 2016, 11:58:10 am
content: Try this?http://www.linuxinternals.org/blog/2014/06/04/a-microsd-card-remote-switcher/As for the suggestion of using multiple SD cards, that should be able to be automated fairly easily too with with an extender cable running to a switchable dock for multiple SD cards. A more elaborate version ofthis.Also, for the Raspi exists Berryboot, which allows you to use multiple image from the same SD card.

author: Jeroen3
author_rank: Super Contributor
timestamp: October 02, 2016, 03:18:56 pm
content: Technically, you could emulate an SD card using ram. Then press reset and it performs fast flash->ram copy to restore it to supposed pre-test state.If you're really into it you could do something like shadow copying to restore even faster. But that obviously is more complicated.However, It all depends if you're able to find code/hdl to function as an SD card memory interface.And, if the costs of developing such automated test equipment is more affordable than a human plugging cards. Which it most likely is not, unless you're going above 4 zero's.

author: encore2097
author_rank: Contributor
timestamp: October 02, 2016, 08:55:36 pm
content: Quote from: abyrvalg on October 02, 2016, 07:13:52 amGotcha, signal source and sink. Thanks.From the spec, the SD Host powers on the SD card, the SD card generates the clock, so I'd need a clock generator.. sounds like there might not be an easy solution other than making an SD card that uses RAM / NFS as storage instead of NAND.

author: encore2097
author_rank: Contributor
timestamp: October 02, 2016, 09:18:06 pm
content: Quote from: mikeselectricstuff on October 02, 2016, 08:34:41 amThere are no pre-verified images, this is during development hence the testing. How this process works: a new build of the firmware is generated after making changes, now we need to see if it works and the performance.1) Write new firmware to SD card2) Attempt to boot device (check bootloader)3) Attempt to boot kernel (check kernel)4) If successful boot, run series of tests/benchmarks and record results (check drivers, application, etc)If the firmware fails at any step prior to 4, there is no way to access the SD card through the device itself. Its like bricking a phone. This is why many embedded devices/SBCs support SD card or similar, which allows you to remove the storage and write to it from a working host.Now to speed up development, its necessary to be able to test multiple builds, on multiple boards simultaneously. It becomes a logistically nightmare to swap 5-10 SD cards every 5-20 minutes for various boards and configurations, while keeping track of what build ran where. Computer vs abacus.Quote from: NottheDan on October 02, 2016, 11:58:10 amThanks for the link. This is another variant of the SD-MUX v1, and has the same issues. This isn't only for RaspberryPi's , I have many other SBCs from different vendors for differing projects. Having to develop/maintain a multi-boot bootloader for each one is not practical and introduces another layer of complexity which gets farther away for testing actual usage.Quote from: Jeroen3 on October 02, 2016, 03:18:56 pmYou would need an RTOS, as the SD interface requires signal responses within a certain time frame and the clock for UHS-I (65+ MB/s) is 100MHz or 208MHz for high speeds. Even then, with such a high clock speed, I don't think this is possible via pure software emulation.. are you aware of something that could enable RAM emulation with these constraints without a micro controller?

author: nctnico
author_rank: Super Contributor
timestamp: October 02, 2016, 09:24:17 pm
content: Quote from: encore2097 on October 02, 2016, 09:18:06 pmYou are doing this wrong! When developing you should be able to access the SD card through the embedded device via the network (sshfs / fuse if it is Linux) in order to keep your sanity. After all the kernel and even the bootloader have to live somewhere on the SD card and should be easy to access. Only program SD cards for production.

author: mikeselectricstuff
author_rank: Super Contributor
timestamp: October 02, 2016, 11:19:57 pm
content: OK I mis-read this as a production-line situation, hence the confusion.As the above post says, seems like you're doing it wrong - I appreciate there may be a very few things that need testing on real cards, but surely once you've loaded some minimum boot-up code from the card, you should be able to handle anything else over a network connection. If you want to make it look as much as possible like an SD card, you write a bootloader that emulates low-level SD commands over the network.Failing that, it ought to be reasonably simple to make a fairly dumb multiplexer between the devices SD port and a USB SD-card reader that means at least you don't need to physically plug & unplug cards.Of course it would be possible to make what is effectively a dual-port, RAM based MicroSD card emulator, it would be a lot of work for what can probably be more easily done in other ways.An FPGA would be able to emulate the SD interface at full speed without too much trouble- you'd probably want a small package located very close to the SD pins for signal integrity - not sure if you'd be able to get full speed extending the pins outside the footprint of the microsd card.The SD interface is command-based, so you could parse the commands into reads and writes to RAM, and you could have a second interface to the RAM - as you already need to do the work to emulate the SD interface, the second interface might as well also be SD as it would simply be a second instantiation of the same HDL component, hooked up to a standard USB SD card-reader IC to give an interface that can go down a cable to the other host.For speed you'd want to emulate using RAM rather than flash, and you wouldn't get enough on an FPGA so would need external DRAM - not a big deal as most FPGAs can talk to DDR2/3If you did decide to go this way, you'd probably end up with something that would be sellable as a product

author: mikeselectricstuff
author_rank: Super Contributor
timestamp: October 02, 2016, 11:35:43 pm
content: Of course an FPGA based hardware emulator like this could also do interesting stuff like logging of SD transactions for debugging and reverse-engineering.

author: andersm
author_rank: Super Contributor
timestamp: October 03, 2016, 07:10:57 am
content: Quote from: encore2097 on October 02, 2016, 05:40:39 amIf that happens, you've got problems in your development process. And in any case, realistically how often will that happen? (The same question applies to your concern about SD card wear - if you have to change cards every couple years, is it a real issue?)

author: tszaboo
author_rank: Super Contributor
timestamp: October 03, 2016, 08:19:54 am
content: Still dont get, why you would need to have to emulate the interface. A normal setup would look like this:You place the board on a "bed of pogopins" test jig. The SD card testpoints make a contact. On the base board you have 2-3-4 whatever number of SD card sockets and a digital multiplexer, controlled by the test jig. You select the SD image you want.If you really want to plug in the SD card jig device, you can still use the multiplexer method.

author: Rasz
author_rank: Super Contributor
timestamp: October 03, 2016, 11:11:02 am
content: Quote from: NANDBlog on October 03, 2016, 08:19:54 amI think the end goal was remote debugging without monkeys swapping cards. Both Wifi SD card and multiple cards + mux + sd reader + some host with ethernet will allow that. Except mux setup will be >$200+time to put it together while Wifi SD is $27 with free shipping on amazon right nowhttps://www.amazon.com/gp/product/B00GEBTFNMhttps://www.amazon.com/gp/product/B01D9JIUU0probably $50 to have it at your doorstep tomorrow.yes, you can upload contents over the wireless interface, here is a dude using wifi SD card in his 3D printer in order to upload prints remotely without touching the cardhttp://www.extrud3d.com/flashair> concerned with performance and the issue of wear and tear on NANDit would take over a year to kill $20 SD card in this scenario (~10MB every 3 minutes for bootloader+kernel+reboot?)instead you are planning to spend couple of hundred on elaborate contraption that will take half a year to develop and debug?

author: mac.6
author_rank: Regular Contributor
timestamp: October 03, 2016, 11:43:15 am
content: For kernel and boot test you can switch to tftp boot instead of using sdcard (if your SBC has native ethernet).You will only need to update sdcard for bootloader and rootfs, and even for this you can do a two step boot (use tftp boot for kernel and rootfs, update SDcard and reboot).

author: CJay
author_rank: Super Contributor
timestamp: October 03, 2016, 02:17:21 pm
content: Can you write the SD over JTAG via the host CPU?Would save all the effort of having to swap cards in and out, just attach your JTAG host to the board under test, either via pogo pins in a test jig or 'standard' header and blast away, if you've concerns over the longevity of the SD cards then you need to be considering the quality of the SD cards.

author: technix
author_rank: Super Contributor
timestamp: October 10, 2016, 01:30:28 pm
content: If your processor support it, you can even leave the card out and boot over JTAG entirely, by holding the processor in RESET, write your bootloader into RAM directly and throw PC there boot straight from RAM. After you get a working bootloader this way (I doubt if a bootloader will get too large to fit in the RAM, and if you need a multi-stage bootloader you can load each stages using JTAG when the card is missing) you can put the bootloader on SD card and start netbooting.

author: vpeter
author_rank: Newbie
timestamp: October 21, 2016, 09:28:18 am
content: Browsing with Google and found this thread...I did similar circuit like this already mentioneda-microsd-card-remote-switcher.But as core IC I tookTS3A27518Ewhich is 6-Bit, 1-of-2 Mux/Demux. Same IC is used in linaro's and tizen's design for automatic testing. Whole circuit works pretty well as prototype. To switch targets I'm using small DPDT switch (not ideal but do the job for manual work). On both sides I'm using flat cable with micro sd card at the end likethis.I would like to make a proper PCB for it but I'm no electronic guyIt should be very small and looking like SD card which would be put in usb sd card writer. On it there is mentioned IC + micro sd card. At the other end there would be flat cable mentioned. If there is someone who is interested in this let me know and I will provide all the details. I'm sure this could be used in a different ways.Peter

author: amyk
author_rank: Super Contributor
timestamp: October 21, 2016, 10:53:17 am
content: Quote from: encore2097 on October 02, 2016, 09:18:06 pmAn SD card basically consists of a microcontroller and NAND flash with firmware. Some are only an 8051, but running at ~100MHz. There's another clump of 8051s and ARMs whose max clock frequency is specified at 208MHz, likely not a coincidence. It's only the communication that happens at such a speed, so they probably have something like a DMA controller and everything else is handled by the firmware.https://www.bunniestudios.com/blog/?p=3554

author: Bruce Abbott
author_rank: Frequent Contributor
timestamp: October 21, 2016, 05:27:05 pm
content: Quote from: encore2097 on October 02, 2016, 09:18:06 pmThat's not the only logistical nightmare. Why would you be creating 5-10 different builds every 5-20 minutes?

author: mac.6
author_rank: Regular Contributor
timestamp: October 21, 2016, 07:59:10 pm
content: This smells like continuous integration server (jenkins/bamboo) with automated builds...

author: rob77
author_rank: Super Contributor
timestamp: October 21, 2016, 08:46:44 pm
content: Quote from: encore2097 on October 02, 2016, 09:18:06 pmi did a boatload of embedded stuff and even custom linux distros (mainly for x86 based network gear) and i can tell you you're not doing it the most efficient way.you have to have a "soft" test-bed for testing builds and once it passes all the regression tests in the soft test-bed then you can go for hardware.... a virtualized test enviroment will boot and run the tests in seconds, compared to minutes when testing on hardware (flash SD, boot HW) .. you can use QEMU for arm on x86 host, and for x86 on x86 host you can use any virtualization software.in general you always have to test in software first, otherwise you'll waste you time... for example back in time when i was writing a network driver kernel module - i was using UML (user mode linux) for the early stage of development.. UML was running an another instance of linux kernel as process on the host linux OS and provided the ideal soft test-bed.. once the HW independent part was OK and stable i moved to the hardware part on real HW. doing that all on hardware from beginning, i would be still doing it (that was approx 10-11 years ago)

author: vpeter
author_rank: Newbie
timestamp: October 22, 2016, 07:18:34 am
content: Sotheyandtheyare doing it also wrong?I really doubt that.

author: rob77
author_rank: Super Contributor
timestamp: October 22, 2016, 03:01:38 pm
content: Quote from: vpeter on October 22, 2016, 07:18:34 amyou should distinguish between different use cases.1. testing software during development - this is much faster in a virtualized environment (soft test-bed)2. testing boards during manufacturing - this can be done with a bunch of SD cardsof course if you have a hardware without a possibility of virtualization then you have to go for testing on hardware where such a SD card switch or even SD emulator has it's benefits.and as always - there are many ways to achieve the same goal.so feel free to doubt anything you wish... but i've been there... just sharing my experience

author: encore2097
author_rank: Contributor
timestamp: November 12, 2016, 05:59:07 am
content: Thanks for all the answers. I ended up with a nifty solution.Quote from: mac.6 on October 21, 2016, 07:59:10 pmSpot on. Though entirely custom, fast, feature-ful and the next step in how hardware and software is built and tested.Quote from: vpeter on October 22, 2016, 07:18:34 amThe folks who worked on this are amazing and great inspiration! I've spoken to them deeply about this problem and while this was the a fast solution at the time, there is much room for improvement.Quote from: rob77 on October 22, 2016, 03:01:38 pmIts quite amazing how many sharp people tell someone they are wrong or doing things "inefficiently" without even understanding the problem. You have made so many assumptions without even attempting to understand my use case.I'm not using x86, but custom RISC hardware. I've already tested in software and trying to emulate in virtualization would be 100x slower than hardware and require FPGAs. e.g. How do you "test" HDMI/LVDS bring up or hardware accelerated network features of your NIC in software?Whats the difference between theory and practice? In theory, there is no difference. If you are not testing on actual hardware, you are shipping your customers lots of bugs.

author: mikeselectricstuff
author_rank: Super Contributor
timestamp: November 12, 2016, 06:32:48 am
content: QuoteBut it can equally be the case that you are utterly convinced you have thought through all the possibilities, but someone else comes at it from a different angle to solve a problem in a different wayyou'd not consideredQuoteMaybe if you'd given more details of your requirements, you'd have got more appropriate suggestions.

author: Rasz
author_rank: Super Contributor
timestamp: November 12, 2016, 09:51:21 am
content: Quote from: encore2097 on November 12, 2016, 05:59:07 amwhat does that have to do with testing bootstrapping?hardware emulation of boot medium gives you precisely ONE advantage - ability to test bootloader. With bootloader tested and working everything else is just network/serial connection transfer away.You dont need to swap(virtually) SD cards to test hdmi. You sound very young and enthusiastic, and quick to dismiss old farts who have been there and done that several times over.

author: 3db
author_rank: Frequent Contributor
timestamp: November 17, 2016, 02:36:58 pm
content: Quote from: Rasz on November 12, 2016, 09:51:21 amAre you an old fart too ?I'm an old fart have you graduated to grumpy old fart yet?

author: il__ya
author_rank: Newbie
timestamp: July 19, 2019, 03:14:42 pm
content: Quote from: encore2097 on November 12, 2016, 05:59:07 amencore2097, sorry for bumping this 3 years old thread, but may I ask what the eventual solution was? I am facing a similar problem at the moment, and would be interested to know how you solved it.

author: encore2097
author_rank: Contributor
timestamp: November 22, 2019, 11:48:54 pm
content: Quote from: il__ya on July 19, 2019, 03:14:42 pmIts unfortunate that I missed this.Here's what I came up with:https://hackaday.io/project/19783-sd-card-emulation

author: coromonadalix
author_rank: Super Contributor
timestamp: November 23, 2019, 06:14:25 pm
content: i recently saw an sd card emulator on Ali for Dwin Hmi displayshttps://www.aliexpress.com/item/32917802827.html?spm=a2g0o.cart.0.0.2c523c00ps3UBV&mp=110,80$ usdand this one toohttps://www.aliexpress.com/item/32830983955.html?spm=a2g0o.cart.0.0.2c523c00ps3UBV&mp=19.90$ usd ??

author: donotdespisethesnake
author_rank: Super Contributor
timestamp: November 24, 2019, 10:01:12 am
content: Those don't emulate an SD card, they are adapters which provide a comms channel from SPI to the USB. The SD card socket is just a way to get to the SPI signals.

author: aldolo
author_rank: Contributor
timestamp: May 26, 2020, 04:26:38 am
content: Any news on this topic? So far I'm considering using 2 real sd card, one on-line and one off-line. the off-line one could be written to using an arduino while the on-line one is used by the final device. then it is easy to swap the 2 with few gates.

author: SimonR
author_rank: Regular Contributor
timestamp: May 26, 2020, 05:22:47 pm
content: Somewhere around 2007/8 I did exactly this more or less.My company developed an ASIC that had an SDIO interface which is effectively an SD card with a peripheral on it instead of flash memory.During development we used an off the shelf SDIO chip until the ASIC (Test FPGA) was ready. Our application was communications device that transferred a lot of data. You could easily have used the chip to implement a memory card, which we were also considering at one point for test purposes.The chip in question was from a company called Arasan and just a demonstration chip for their IP which I think we used. Its not available any more but Arasan seem to be going stillhttps://www.arasan.com/products/Alternatively while I was searching for the Arasan chip I also found this from Latticehttp://www.latticesemi.com/en/Products/DesignSoftwareAndIP/IntellectualProperty/IPCore/EurekaCores/EP560SDSDIOMMCSlaveControllerThis looks like it might be what everyone is looking for.Just a suggestion

author: ethaniel
author_rank: Newbie
timestamp: October 09, 2021, 12:19:40 pm
content: Sorry for bumping up an old thread. I'm willing to set a $2K bounty on this if anyone is interested in exploring the idea.I'm trying to find a way how to capture data that's being written to a microsd card and receive it live on linux.I've got a GoPro MAX which can record 360 degrees video. It doesn't have an HDMI output.I want to extract the h264/h265 video data as it's being written into files on the "virtual" microsd card.I assume there is alot of wizardry involved - letting the gopro know that this it's a working sd card with an actual filesystem and free space available. Perhaps somehow patch it through to a virtual block device in linux? There is a lot of data transfer involved (100+ megabytes per second).My goal is to use feed the raw h264 into a streaming program, so I could use the GoPro as a 24/7 live camera.Anyways, the bounty is $2K. Looking forward to hearing back from you.