Some thoughts on PCI on 3.3V FPGAs - Page 1
Forum section: electronics / general

author: Forty-Bot
author_rank: Contributor
timestamp: January 22, 2023, 06:37:13 am
content: The other thread about PCI reminded me of my recent thoughts on this subject. I'm going to write some of them down before I forget.PCI has some nice features. It's slow enough that you don't need serdes or other fast I/Os, so it can be used on cheap/slow/FOSS FPGAs. You can use it on a regular PC, no need to add a soft-cpu to control things. And there's a ton of old motherboards with PCI slots. Unfortunately, there are some problemsFirst, while there's a 3.3 V version of the bus, almost no motherboards implement it. So you're stuck using 5V. This means either using a 5 V FPGA (which don't get made any more and have outdated tooling) or level shifting. The main problem with level shifters is that the delay will affect your timing. You have 11 ns from when the clock goes high to when your output has to be valid, and 7 ns before the clock goes high when inputs are valid. Depending on your FPGA, you will lose several ns just waiting for your signals to get off-chip (or on-chip). This can be offset a little using PLLs, but then you start to eat into your input setup time. So ideally, you want very fast level shifters, preferably with sub-5 ns propagation delays. Fortunately, even faster level shifters exist. "FET Bus switches" like the CBT or CB3T families (and several others from different vendors) connect the in/outs directly to the source/drain of a FET, with the gate held at a constant voltage. By keeping the gate at 4.3 V or so, you can prevent either side from ever getting above 3.3 V, which is a logic high for 5 V TTL. These level shifters typically have propagation delays on the level of 250 ps, due to RC delay, which is effectively instant. We end up with around 10ns to decide what to do.Unfortunately, there's another problem with level shifting. You can have a maximum of 10 pF on any input pin. Around half of that is going to be eaten up by your FPGA. Unfortunately, those bus switches from earlier have a capacitance of 10-15 pF when on (aka functioning), which eats up the other three halves. On top of this, reflections on the bus can cause signals to spike briefly to 11 V or -5.5 V, requiring diodes for protection (the level shifters will typically have clamping diodes, but you need something more sturdy). Youcanfind diodes with suitably low capacitance, like the 1PS302 or BAT63-07W, but they don't usually come in bulk packages, so you'd have to have one per signal. Fortunately, the capacitance is per-slot, so you can always use a multi-slot motherboard and leave the other slot empty.The last electrical problem is R_ON. Although not explicitly spelled out in the spec, you'll want to have reasonable R_ON in order to meet the driver requirements, especially since bus switches provide no amplification. Unfortunately, R_ON depends on V_GS, which is capped by the need to perform level shifting. The best I found was around 8 Ohms (e.g. TC7MPB9307FK).There are also some trace length limits (<1.5" for signals, and =2.5" for the clock). Presumably these are specified to ensure proper timing, but I'm not sure how everything interacts.In terms of timing, while you can take a few cycles for address decoding, and you can wait up to 8 cycles between transfers (and up to 16 on the first cycle), you still have to respond to inputs on the previous cycle for IRDY/TRDY handshaking. This isn't terribly onerous, but you'll certainly have better throughput if you can respond faster.So you can do everything with level-shifting, but it won't completely be in-spec (due to capacitance). What are some other options?You can add a PCI-to-PCI bridge, or better yet a PCIe-to-PCI bridge. This lets you work at 3.3 V and skip the level shifter and protection diodes. This is probably the most practical way.You can use a PCIe PIPE phy, like the XIO1100. This implements around 2/3 of the PCIe physical layer, so you'd need to do the other 1/3 of the physical layer, the data link layer, and the transaction layer handling on the FPGA. There are some projects implementing the transaction layer, but I couldn't find anything doing the data link layer. PIPE is a 16 bit bus running at 125 MHz, which is much faster than PCI, but not much faster than the 10 ns (aka 100 MHz) you'd need to work at anyway. There aren't any back-to-back responses, and the XIO1100 will align the clock to the center of the data, so it's not all that bad.

author: DiTBho
author_rank: Super Contributor
timestamp: January 22, 2023, 04:20:35 pm
content: Quote from: Forty-Bot on January 22, 2023, 06:37:13 amon my HP RISC workstationsCode:PCI-32/6666 MHz32-bit266 MB/s3.3 VPCI-64/3333 MHz64-bit266 MB/s3.3 V/5 VPCI-64/6666 MHz64-bit533 MB/s3.3 VPCI-X100 MHz64-bit800 MB/s3.3 Vmy MIPS router has miniPCI, 32-bit 3.3V data @33Mhz.PCI-64 3.3V is perfect: it can be downgraded to 32-bit

author: DiTBho
author_rank: Super Contributor
timestamp: January 22, 2023, 04:21:46 pm
content: Quote from: Forty-Bot on January 22, 2023, 06:37:13 ammytopic

author: berke
author_rank: Frequent Contributor
timestamp: January 22, 2023, 06:16:49 pm
content: Quote from: Forty-Bot on January 22, 2023, 06:37:13 amThat's quite interesting, do you have a link to a PCIe-to-PCI adapter that works at 3.3V? It would be surprising if those are readily available if 3.3V PCI boards were a very tiny minority.But it would be a very neat solution for all the cases where you want to pipe a data stream from a cheap/FOSS FPGA to something with enough RAM and with good reliability. The signals are also slow enough that you have a chance at scoping or analyzing them without having to sell any kidneys.

author: Forty-Bot
author_rank: Contributor
timestamp: January 22, 2023, 07:02:54 pm
content: Quote from: DiTBho on January 22, 2023, 04:20:35 pmYeah, that's certainly the way to go if you have it. But it's much less common on PCs, as they all want to support old 5V-only cards. For example, aquick searchshows no motherboards on the first page with 3.3V PCI slots.QuoteMost bridges will have a dedicated input rail for the I/O supply, so it's pretty easy to select 3.3V or 5V as you please. For example, PEX8311-AA66BCF, PI7C9X111SL, and XIO2001 all support 3.3V PCI busses.

author: dolbeau
author_rank: Regular Contributor
timestamp: January 23, 2023, 05:36:37 pm
content: If you want to create a device for a PCI host (using a PCIe device), don't you need a PCI-to-PCIe bridge rather than PCIe-to-PCI? Some PCIe-to-PCI are 'reversible' like the PI7C9X110, but I don't think all of them are? Also the number of available SKUs is dwindling :-( For PCI-to-PCI, Mouser pretty much only has the PCI2250P left.I can confirm CB3T works fine for at least some vintage bus - I use them (after some help from this very forum!) to put anArtix-7 (so 3.3V) on SBus (5V), Sun's expansion bus they moved to PCI. Works fine, but SBus has a capacitance limit of 20pF per signal (more for some) and is up to 25 MHz so it's got a bit more margin than PCI.Though with this talk of reflections and spike (I'm a SW guy...), you now got me worried about long-term effects, I don't have any clamping on the bus - just the CB3T standing between the bus & the FPGA...

author: berke
author_rank: Frequent Contributor
timestamp: January 23, 2023, 05:43:11 pm
content: Quote from: dolbeau on January 23, 2023, 05:36:37 pmThat's the thing, with a bridge you won't need a PCI host, you can use a modern device.

author: dolbeau
author_rank: Regular Contributor
timestamp: January 23, 2023, 06:04:17 pm
content: Quote from: DiTBho on January 22, 2023, 04:20:35 pmOut of curiosity, which one? I recently 'inherited' a C240 from a friend, I investigated the GSC slots (publicly available information: none) but forgot to check the voltage on the PCI slots - you just made me realize it had 3V3-only slots (for the 64b/66MHz)! So in theory, they could interface directly with an Artix-7. Might be a solution to add some devices to the C240 like the SBusFPGA does to my SPARCstation 20.Annoyingly, PowerMac G4 with 64b/66MHz slots (on any other) are 5V-only, so they won't take any 3V3-only card, which most PCI-X cards are (I have never seen a 5V SAS controller, for instance). I discovered that when I wanted to try a PCI-X LSI1064 device flashed with a Sun F-ROM to see if it would boot the G4. A PCIe LSI1068e flashed the same way did boot a PCIe G5 (I don't have a PCI-X G5 to test the theory further).Incidentally and more on-topic, the 'bridge' solution is almost as old as PCI - early PCI Sun's (like the Ultra 5 or Ultra 10) had 5V-only PCI bus, but later ones like the Blade 100 were 3.3V-only, with a PCI-PCI bridge to create backward-compatible 5V slots.And ultimately the real issue with a new PCI device will turn out to be how to support the thing in firmware and software...

author: DiTBho
author_rank: Super Contributor
timestamp: January 23, 2023, 06:46:23 pm
content: C3600

author: asmi
author_rank: Super Contributor
timestamp: January 23, 2023, 06:58:51 pm
content: Quote from: dolbeau on January 23, 2023, 06:04:17 pmIf you have Artix, you might as well simply implement a PCI Express, it's going to be much easier because that FPGA contains both transceivers and PCIE block, so all you need to do is to process packets, the rest is taken care of by the hardware. But if all you want is just to map a part (or whole) of device's memory into the host's address space and have some register interface in a PCI config space, there are free IPs from Xilinx which do just that.Quote from: dolbeau on January 23, 2023, 06:04:17 pmThat's actually the easier part, because PCI/PCIE devices typically look like any other memory-mapped device to a CPU once the BIOS/UEFI has done it's job and performed the mapping. Of course if your device is complex, so will be your driver, but in general I think complexity of PCI/PCIE is a bit overstated in many cases when it comes to mainsteam OSes like Linux and Windows, because they come with the code to handle a lot of low-level details for you.

author: dolbeau
author_rank: Regular Contributor
timestamp: January 23, 2023, 07:47:36 pm
content: Quote from: asmi on January 23, 2023, 06:58:51 pmBut if the host only has PCI (C-class PA-RISC workstation like my C240 or DiTBho's C3600 are from the end of the 90's), then either you implement PCI or you need a PCI-to-PCIe bridge to the Artix. And DiTBho's original thread is about the MIPS Atlas, an old devboard for MIPS cores - also only PCI, but apparently with 3.3V support.Of course is the FPGA itself was to be the primary host, then PCIe is the way to go on Artix. In fact I've been thinking about implementing a SBus <-> PCIe bridge, but that would require a PCIe root complex rather than an endpoint in the Artix. Xilinx' IP is an option, but open-source would be better; Litex already has PCIe endpoint, unfortunately not (yet?) the root complex. Also the Xilinx IP have driver in Linux, but not in NetBSD that I know of (lots of old hardware is not/no longer supported by Linux but still by NetBSD).(PCI-not-express is likely of no interest to anyone not involved with vintage systems anymore. PCIe is the way to go in the modern world, but this thread started specifically about PCI-not-express, such as the 5V stuff).QuoteYou do need some support in the firmware - in particular is they are required for booting like non-volatile storage or networking - which may not be BIOS/UEFI. That's not always easy. On my SPARC systems it's OpenBoot (later to become OpenFirmware, also available on a lot of PowerPC systems) using Forth. For PA-RISC workstation, no idea what is needed. And if the devices are not already supported in the OS (in which case you need just the bus glue, done that for USB OHCI on SBus), you also need some drivers there. In my experience, getting all those bits together is harder than the actual hardware, as a lot of the knowledge and expertise to program in obsolete environments is gone (again - for PCI-not-express and older busses).Quote from: DiTBho on January 23, 2023, 06:46:23 pmNice toy, lucky you :-)

author: asmi
author_rank: Super Contributor
timestamp: January 23, 2023, 08:12:37 pm
content: Quote from: dolbeau on January 23, 2023, 07:47:36 pmPCIE block in Artix can be configured as a root complex as well as an endpoint.Quote from: dolbeau on January 23, 2023, 07:47:36 pmIt's a hardware block, so it can't be open source unless you don't mind wasting a bunch of FPGA resources to reimplement functionality of that hardware.

author: nctnico
author_rank: Super Contributor
timestamp: January 24, 2023, 01:27:07 am
content: PCI works just fine with 3.3V signalling. 20+ years ago I have designed PCI interface cards using 3.3V FPGAs (IIRC Xilinx Spartan-II and Virtex 2). These have been produced in large quantities and have been used in many different PCs. These cards where keyed to fit both 3.3V and 5V slots. These FPGAs are 5V tolerant though.

author: Forty-Bot
author_rank: Contributor
timestamp: January 24, 2023, 01:49:34 am
content: Quote from: dolbeau on January 23, 2023, 05:36:37 pmWell, the whole point is to get away with using an FPGA that is too slow to have a serdes capable of PCIe.QuoteIt was your post on these guys that inspired me to look into them ))QuoteThe CB3T has internal clamping diodes, they just aren't spec'd for the PCI worst case. So if anything fails, it should be your level shifter and not your FPGA. That said, the PCI specs are definitely worst-case, and probably will never happen in the real world (especially if using newer PCI devices).Quote from: nctnico on January 24, 2023, 01:27:07 amYeah, it's much easier to find 3.3V cards vs 3.3V hosts. The problem with using 5V-tolerant FPGAs of course is that just like PCI they are a dying breed.Quote from: dolbeau on January 23, 2023, 07:47:36 pmBooting from a PCI device is a while level beyond what's necessary just to get something working.

author: DiTBho
author_rank: Super Contributor
timestamp: January 24, 2023, 09:56:06 am
content: PCI-X should be 3.3V-only.Perhaps useful if you can slow down the clock to 33Mhz

author: dmendesf
author_rank: Frequent Contributor
timestamp: January 25, 2023, 02:13:36 am
content: Two other avenues for work:WCH has some generic glue pcie bridges (o mean pcie to some simple synchonous bus)http://www.wch-ic.com/products/category/2.html#dataThere are some ready made pcie-to-pci bus converters. As there's no 5V in the pcie slot i presume It gets 5V from the Power cable. Maybe you can hook It to 3V3 and be happy:https://a.aliexpress.com/_m0zLdPs

author: dolbeau
author_rank: Regular Contributor
timestamp: January 25, 2023, 07:41:03 am
content: Quote from: asmi on January 23, 2023, 08:12:37 pmThanks for the comments - they led me to investigate how Litex does it. It turns out for 7-series (and some others presumably) they just create a nice wrapperaround the hard block. So it should be possible to do the same for root complex mode, and benefits from Litex ease-of-use & Xilinx' IP...

author: Forty-Bot
author_rank: Contributor
timestamp: January 25, 2023, 09:57:47 pm
content: Quote from: dmendesf on January 25, 2023, 02:13:36 amAh, these are interesting. CH368 works out to 44 MiB/s (32 bits @ 11 MHz), and it's done all the bus decoding already. It says it supports DMA, but there's no mention of how it's implemented. Acouplepeoplehave also wondered about this. I had a look attheir driver, but even though they enable pci_master, they don't seem to use it.While looking around, I also foundthis thread, which talks about PLX PCI9303/PCI9054 chips. These seem great, but they are quite expensive now that they are no longer produced.

author: DiTBho
author_rank: Super Contributor
timestamp: January 26, 2023, 04:10:40 pm
content: PCI to ISA, but then it's 5Vwhat about Cardbus? PCMCIA?

author: DiTBho
author_rank: Super Contributor
timestamp: January 26, 2023, 04:45:42 pm
content: What about AGP?AGP is a heavily modified PCI 3.3V slot.Some UNIX workstations (e.g. HP C8K) have AGPs as well as some x86-PCs.

author: DiTBho
author_rank: Super Contributor
timestamp: February 02, 2023, 11:16:24 pm
content: anyone?