Testing of PCIe boards in manufacturing - Page 1
Forum section: electronics / projects

author: motocoder
author_rank: Unknown
timestamp: July 03, 2014, 05:34:29 pm
content: Hi All -My company is about to start a production run of a board with a PCI-Express interface. For our initial manufacturing run, we're having the manufacturer insert each board in a computer we provide. The computer will boot into a lightweight OS image that will then trigger some tests on the board and will stress test the PCIe interface by sending data back and forth to the board over PCIe.While this process works, it's extremely slow due mostly to the boot times of the computer. For reasons I won't get into here, we actually have to boot the computer twice to test each card. The slow test time directly translates into increased manufacturing costs. For future production runs, which will be larger volume, we want to find a more efficient way to test the boards.So I am wondering if others here have any insights they can share as to how testing of PCIe cards is handled by companies that mass produce PCIe boards.Thanks

author: AndyC_772
author_rank: Super Contributor
timestamp: July 03, 2014, 06:12:24 pm
content: Have you looked into using an active extender card that would allow the board to be hot swapped?

author: SirNick
author_rank: Frequent Contributor
timestamp: July 03, 2014, 06:50:23 pm
content: Yeah. Especially if you can use something like Linux's modprobe to load and unload the necessary kernel drivers, hot-swap should be an option.

author: motocoder
author_rank: Unknown
timestamp: July 03, 2014, 06:54:03 pm
content: Quote from: AndyC_772 on July 03, 2014, 06:12:24 pmI wasn't even aware of such a thing. If we do go that route, though, how do we get the OS to recognize the card change? We are currently using WinPE, and support for hot swap in windows seems to be lacking. Does it work we'll in Linux?

author: rob77
author_rank: Super Contributor
timestamp: July 03, 2014, 11:45:55 pm
content: back in time in my old job we were testing the PCI cards ourselves , the manufacturer just assembled the boards and loaded the firmware for the fpga (it was a low volume production - 100 cards every 2-3 months)we were simply doing it by testing 5 PCI cards at once in a single mainboard - it took only a few hours to test the whole batch.

author: SirNick
author_rank: Frequent Contributor
timestamp: July 03, 2014, 11:54:12 pm
content: Quote from: motocoder on July 03, 2014, 06:54:03 pmIn theory, it should. From my limited experience in kernel hacking, the kernel is written in such a way to make enumeration of PCI not too unlike other buses where things come and go with regularity. It works quite well for USB (as you'd expect), and SATA (which is perhaps not quite as obvious). I have never had the occasion to swap PCI cards in situ, but if your driver can be ported to Linux, it would certainly be worth a shot.

author: mikeselectricstuff
author_rank: Super Contributor
timestamp: July 04, 2014, 12:10:24 am
content: Could you hibernate instead of reboot?

author: jmole
author_rank: Regular Contributor
timestamp: July 04, 2014, 12:35:53 am
content: Along with the other suggestions, you might consider using a computer (laptop?) with a thunderbolt port and an external PCIe chassis for testing.The less "dicking around" the test folks have to do inside a working computer (barebones or not), the better.The suggestions about Linux, hotplug, and modprobe are all worth looking into as well.I know you didn't want to discuss it in depth, but why the dual reboot process?

author: mikeselectricstuff
author_rank: Super Contributor
timestamp: July 04, 2014, 12:50:59 am
content: Could you at least test more than one board per reboot, i.e. have several inserted and test one at a time ?

author: Stonent
author_rank: Super Contributor
timestamp: July 04, 2014, 12:52:50 am
content: I have successfully swapped PCI cards during a hibernate under windows. It acted just like you had plugged in a USB device.I've also hibernated a computer, installed additional DIMMs and brought it out of hibernate. It didn't detect the ram until the reboot but I was dealing with someone who kept giving me all these times he was available that were ludicrous.Another thing, if boot times are a problem, get a solid state drive. Trust me, they help immensely with boot times.

author: Stonent
author_rank: Super Contributor
timestamp: July 04, 2014, 12:56:24 am
content: Another thought. There are Expresscard to PCIE adapters, the one's that I've seen so far are just 1x but PCIE doesn't need all the lanes in order to function. You could cut the plastic out of the back and just have the card sticking out.Now if you need to test all lanes, that's a different story.There are also PCIE hotswap boards.http://www.quarch.com/products/qtl1630.html

author: motocoder
author_rank: Unknown
timestamp: July 04, 2014, 03:57:59 am
content: Quote from: rob77 on July 03, 2014, 11:45:55 pmI think we will definitely look into multiple cards per host PC / test iteration. That seems like a very simple extension that will help quite a bit.Quote from: SirNick on July 03, 2014, 11:54:12 pmI spent today trying to test it out. Boot into a Linux LiveCD on a USB stick with an empty FPGA (no PCIe), program an image with PCIe, and see if the kernel detects the device. However, I ran into some compatibility issues between Linux and the crappy HP PC I have at work. I'll have to repeat the test on another PC.Quote from: mikeselectricstuff on July 04, 2014, 12:10:24 amI don't think WinPE supports hibernate.Quote from: jmole on July 04, 2014, 12:35:53 amThe PCIe interface is implemented via an FPGA. You program the FPGA to enable the PCIe interface. We do this via an on-board USB-based JTAG interface, but neither WinPE nor Windows will pick up the PCIe device w/o rebooting the machine. So we have to boot, check the FPGA program signature via JTAG, program the FPGA, and then boot a second time before we can start the PCIe testing.Quote from: mikeselectricstuff on July 04, 2014, 12:50:59 amYes, rob77 suggested this as well. It's a good idea, although there are typically only a few x8 slots in a PC, so it may limit how many we can do concurrently to 2 or 3.Quote from: Stonent on July 04, 2014, 12:52:50 amI thought so too, but I tested that yesterday and it only shaved about 15 seconds off a 2:25 boot time (most of that time is in the BIOS, and we expect to reduce that quite a bit).Quote from: Stonent on July 04, 2014, 12:56:24 amWe definitely need to test all lanes. Thanks for the link, a hotswap board would be useful combined with the Linux hotswap support discussed earlier in the thread.

author: peter.mitchell
author_rank: Super Contributor
timestamp: July 04, 2014, 07:58:21 am
content: could you program the FPGA boards externally, as a separate setup/assembly stage to cut out one of the reboots?

author: Jeroen3
author_rank: Super Contributor
timestamp: July 04, 2014, 11:56:22 am
content: QuoteYou're joking?Yesterday I booted a live lubuntu in less than 20 seconds from a USB 3.0 drive on a efi board.You should ditch windows for production testing. But if you can't, you can have 5 test systems in parallel or something.That will be a good idea anyway, because what happens when a bad card damages your tester. Failed deadline?

author: motocoder
author_rank: Unknown
timestamp: July 04, 2014, 12:22:41 pm
content: Quote from: Jeroen3 on July 04, 2014, 11:56:22 amBoot time for a Linux LiveCD is significantly longer than the times I quoted. It's mostly BIOS boot time, not the OS.

author: nctnico
author_rank: Super Contributor
timestamp: July 04, 2014, 03:41:20 pm
content: Then tweak the BIOS to be quicker or use a machine with a faster BIOS. Then again PCIe can be hotswapped with the proper hardware so it's best to go that route. Also make sure to have extra motherboards. PCIe connectors are rated for 10 to 30 mating cycles.

author: Sebastian
author_rank: Regular Contributor
timestamp: July 04, 2014, 04:30:22 pm
content: If the number of PCIe slots is a problem, look at gaming mainboards. Four x16 slots should be easy to find, maybe more. I think you said that you need only 8x, I don't know if there are riser cards or something to make two x8 out of one x16, but if it is posible you could test 8 boards at once minimum.

author: AndyC_772
author_rank: Super Contributor
timestamp: July 04, 2014, 05:23:41 pm
content: Quote from: nctnico on July 04, 2014, 03:41:20 pmThis is a good point, the connectors do wear out very quickly in a production environment.To extend the life of the test rig when I was testing ISA and PCI cards, I used to add a short (passive) extender in between the active riser card and the UUT, so it was that card's connector that wore out and not one that's actually attached to something worthwhile.When cards started failing, I'd swap out the passive extender for a new one.Not sure if that's really an option with PCIe, though. Frequencies have gone up since I last work in a production test environment.

author: rob77
author_rank: Super Contributor
timestamp: July 04, 2014, 06:11:52 pm
content: i don't know how durable are the PCIe slots, but in our case the 32bit PCI slots were lasting 100s of cycles (it was approx 9-10 years ago). but nowadays thanks to the excessive cost reduction everywhere, it easily might be only 10s of cycles

author: SirNick
author_rank: Frequent Contributor
timestamp: July 08, 2014, 10:22:52 pm
content: Quote from: AndyC_772 on July 04, 2014, 05:23:41 pmShould still be possible. Look at embedded computer suppliers (e.g., Logic Supply) for low-profile case brackets. Some are just 90-deg PCBs, but others have short, shielded ribbon cables. That would reduce the wear and tear on the mobo slots by an order of magnitude.

author: Phaedrus
author_rank: Frequent Contributor
timestamp: July 08, 2014, 10:50:37 pm
content: I've helped quite a few people to speed up boot times of consumer PCs. Windows 7 on an SSD can boot in ~15-20 seconds including BIOS/UEFI start. Lightweight Linux on an SSD can boot in ~8-10 seconds.Try disabling all unused controllers in BIOS. Legacy IDE controllers alone often take 10-20 seconds to POST. Disable USB and networking if you don't need them; each one will shave a second off your boot time. If your board has multiple SATA controllers, experiment to see which boots the fastest. Most importantly, put your OS on an SSD or a damn fast USB thumb drive. If this is an ATE and won't be running additional software, something like this should suffice:http://www.newegg.com/Product/Product.aspx?Item=N82E16820211595Not the fastest, but fast enough to boot Linux in 3-5 seconds.This combined with testing multiple boards at once should get you up to a pretty respectable speed.

author: Artlav
author_rank: Frequent Contributor
timestamp: July 09, 2014, 12:29:46 pm
content: About boot times - you can try getting a motherboard that supports CoreBoothttp://www.coreboot.org, an open-source lightweight BIOS replacement.That one can boot from power off to Linux in a few seconds.

author: ConKbot
author_rank: Super Contributor
timestamp: July 09, 2014, 01:04:37 pm
content: Quote from: AndyC_772 on July 04, 2014, 05:23:41 pmThe good news is that PCIe isnt a bus with a bunch of stubs, but point to point. With PCI, if you got your extenders too long then reflections from stubs, etc can really slow down the bus or make stuff just stop working. with a point to point link like PCIe, then as long as the extension maintains good signal integrety (proper impedance when routing differential pairs, etc) and the signal levels are still adequate, the extension should be fine. The bad news is that its microwave frequencies, so the extender cant be absolutely dirt cheap.

author: AndyC_772
author_rank: Super Contributor
timestamp: July 09, 2014, 04:52:44 pm
content: That's what I was getting at; an extender card would need to be carefully impedance matched and would almost certainly require active drivers and receivers in order to work correctly. I guess there's no reason why an extender card couldn't, or shouldn't, have these.

author: nctnico
author_rank: Super Contributor
timestamp: July 09, 2014, 05:25:23 pm
content: Actually you don't need/want repeater on a PCIe extender card. IIRC I have seen PCIe hardware which is connected to the motherboard using cables. I wouldn't be surprised if you can get PCIe to work using a short (50cm) HDMI cable.

author: AndyC_772
author_rank: Super Contributor
timestamp: July 09, 2014, 05:47:55 pm
content: It's not hot swap capable, though... wouldn't you want to isolate the PCIe bus when the board is powered down?With that in mind, is there not also a risk that the bus isolator and extra connectors in the middle of the bus would constitute an impedance discontinuity?

author: nctnico
author_rank: Super Contributor
timestamp: July 09, 2014, 06:44:03 pm
content: PCIe is hotpluggable by default. The lines are capacitive coupled so the signal is AC only. Remember PCIe is a several Gbit/s point to point serial link. As long as impedances are properly matched and losses are limited everything is fine. If the connectors have the proper impedance then the signal does not see a discontinuity.