PCIe lane switching - Page 1
Forum section: electronics / beginners

author: QinX
author_rank: Contributor
timestamp: March 13, 2015, 10:06:30 am
content: Hi everyone,New to the forum, but I've been watching EEVblog videos for a while now.Unfortunately I've gathered enough knowledge to be feared by most of the electronics around me, but I still have a long way to go.The other problem is, I have ideas that I shouldn't really be trying with my current knowledge.So what I want to do is:build a PCB that switches a PCIe 3.0 16x lane into 2x slots that are PCIe 3.0 8x.It is done by motherboard manufactures for a while already.If I understand correctly they use ASMedia ASM1480 switches to switch 4 lanes per chip.https://dl.dropboxusercontent.com/u/15977794/ASM1480_Data_Sheet_R1_4.pdfSo if 1 card is inserted all chips are switched to send the PCIe lanes to the first slot, giving is the full 16x bandwidth.If a second card is inserted 2 of the cips are switched to send a total of 8 PCIe lanes to the second slot, giving each card half the bandwidth.Now if this is true and I want to build a PCB that has 2 slots that are 8x do I actually need these chips?Couldn't I just route the first 8 lanes to the first slot and the back 8 lanes to the second slot?I will always have 2 cards installed so I wouldn't need to "optimize" bandwidth allocation.I've already found this PCIe Riser card which doesn't really seem to have any special componentsIt mentions on the PCB that Slot 3 is PCIe 4x, so it actual splits the incoming 16 lanes into 8/4/4 I guessBut this one has a lot of componentshttp://www.acmemicro.com/Product/13546/Supermicro-RSC-R2UG-A2E16-A-LHS-Active-PCI-E-2U-Riser-Card?c_id=356but why does it have some many components? it even has a chip that needs cooling(might be a PLX chip), PLX allow for 16X input and 2x 16X output, magic.I hope you can assist me in figuring out what can actually be done in terms of splitting PCIe lanes.Regards,QinX

author: QinX
author_rank: Contributor
timestamp: March 13, 2015, 11:36:09 am
content: I've done some more research and there seems to be some kind of catch if you want to split an existing 16x lane to 8x/8xIt is called PCIe Bifurcation and it seems to be a feature that needs to be implemented in the BIOS.http://www.intel.com/content/www/us/en/intelligent-systems/maho-bay/core-i7-pcie-slot-bifurcation-demo.htmlHowever it think I should still be able to split the lanes even without it being implemented int he BIOS, but I might be need components for that.I'm having a hard time finding exact information about it.One of the things I came across is this Stackexchange post about the clocks is that you can't just split it.http://electronics.stackexchange.com/questions/93264/pci-express-bifurcation-clock-fanout-buffer-needed-to-split-reference-clock

author: Mechanical Menace
author_rank: Super Contributor
timestamp: March 13, 2015, 02:51:15 pm
content: You can only configure how the lanes are split by modding your MoBo, if that's even possible. In the video you linked to they're telling MoBo designers how they can set pins for different configurations that their board supports...Quote from: QinX on March 13, 2015, 10:06:30 amThey're multiplexers, they don't let you switch the lane configuration on the fly, they let you switch between different devices on the same bus. So you can have say more than 1 PCIE 16 device hooked up even though there's only one x16 connection to the CPU. You use the multiplexers to decide which device you;'re talking to on the same serial bus.

author: German_EE
author_rank: Super Contributor
timestamp: March 13, 2015, 02:56:31 pm
content: You have forgotten about one problem, timing. Given the speed of operation of PCIe you will need to make sure that a number of signals arrive at the connector at PRECISELY the same time, this is a significant exercise that may require a multi-layer PCB.

author: macboy
author_rank: Super Contributor
timestamp: March 13, 2015, 03:10:10 pm
content: The second example card you have shown likely uses a PCIe-PCIe bridge. While the BIOS must support such a bridge in order for the connected cards to be seen at boot time, I don't think this is generally an issue. I would not be surprised if most BIOS do support them, since it is a common function. SATA3/USB3 combo cards for example usually have such a bridge. I have also seen a high performance RAID card with a PCIe bridge to enable a PCIe gen 2 chipset to have a fast interface to a PCIe gen 1 motherboard in an x4 slot.

author: Alex Eisenhut
author_rank: Super Contributor
timestamp: March 13, 2015, 03:18:46 pm
content: Quote from: German_EE on March 13, 2015, 02:56:31 pmNot really, PCIe does link training and has a lot of leeway, you can even switch + and - on a lane and it will adapt. You only need to match the +/- of a single pair, the pair to pair is not that critical.I don't get paid to deskew PCIe layouts, it's never happened. DDR3, OTOH...

author: Monkeh
author_rank: Super Contributor
timestamp: March 13, 2015, 03:42:07 pm
content: Quote from: German_EE on March 13, 2015, 02:56:31 pmPCI-E does not require pair-to-pair matching. It is routed as a group of independent differential pairs. It's actually relatively easy because of this.

author: BlueBill
author_rank: Regular Contributor
timestamp: March 13, 2015, 03:43:24 pm
content: With the amount of skill, time & equipment you'd need, it would be far far cheaper just to buy a motherboard that does what you want.PS why would you want so many PCI slots? A bitcoin miner?

author: QinX
author_rank: Contributor
timestamp: March 13, 2015, 06:25:19 pm
content: Everyone Thank you for the replies, I'm hoping we can figure out how to make it work.I could email a motherboard manufacturer about this, but I wouldn't know how to get my message to the right person, It'll most likely get picked up by the regular tech support.Quote from: Mechanical Menace on March 13, 2015, 02:51:15 pmI'm not looking to switch lanes on the fly, I already figured they wouldn't work like that, however what they do seem to do is either allow the 2 slots they are connected to to be either 16x/0x ,just one card installed, or 8x/8x , two cards installed.Note I'm talking about how these chips are implemented on motherboards, they are used in sets of 4.What I'm interested in knowing is if I use these chips on a separate riser PCB that plugs into a 16x slot, if I could make it 8x/8x on the riser PCB.Quote from: German_EE on March 13, 2015, 02:56:31 pmThe Riser PCB that I want to make plugs into a motherboard that already has all the clocks and regulators on board. I only want to split the signals.I already made a Riser PCB that goes from 4x on the board to 4x on the riser. The differential pairs don't need to match in length. However the 2 traces of each differental pair need to be within a certain length, I believe it was 8mil.Quote from: macboy on March 13, 2015, 03:10:10 pmI've seen a so called PLX PEX8747 be used, however this chip doesn't split the lanes, it actually multiplies them, so you input 16x and get 32x out. These are power hungry, >7Watts, and are $$$. around 80 a piece, that is a no go for me.Quote from: BlueBill on March 13, 2015, 03:43:24 pmI can't buy a motherboard that does what I want. I'm working on something special based on a M-ITX board.A M-ITX only has 1 PCIe slot, often 16. However a PCIe 8x slot has plenty of bandwidth for 1 GPU. So why not have 2.Mini DTX boards have 2 slots, but the boards out there you can count on 1 hand.

author: Mechanical Menace
author_rank: Super Contributor
timestamp: March 13, 2015, 06:29:48 pm
content: Quote from: QinX on March 13, 2015, 06:25:19 pmThey're 4 lane multiplexers, so you need 4 working in parallel to switch between 2 16 lane devices, 2 for 8 lane, 1 for 4 lane...

author: QinX
author_rank: Contributor
timestamp: March 13, 2015, 06:57:29 pm
content: Quote from: Mechanical Menace on March 13, 2015, 06:29:48 pmI'm not so sure.If you take a really good look at the first picture at the top, the top PCIe slot is physical 16x and signal 16x. However, the second Red PCIe slot is physical 16x and signal 8x. It only has the pins inside it to support 8 lanes. But they have 4 ASM1480 on the board between them.So why use 4 ASM1480, capable of 16x multiplexing if you are running both slots at 8x/8x?Hmm I've looked this one up,MASSIVE IMAGE:http://images.anandtech.com/doci/8449/Project_03%20.jpgYou can see how the second red and fourth red slot are wired only for 8x, but here they have 8 times the ASM1480, but I guess they have the 4 needed for the bottom slot there as well, must have been easier to route.EDIT:I figured it outYou have a RX and TX Pair, so there are actualy 32 pairs total. and for 8x you still have 16 pairs total. So the ASM1480 is purely used to switch 16 of the 32 pairs to the other slot.I do think it is a hard switch. probably when a second card is detected the selection port goes high on the chips and switches 16 pairs of to the other slot.Now would the system need to be aware of the 2 PCIe slot or will it magically work?

author: Mechanical Menace
author_rank: Super Contributor
timestamp: March 13, 2015, 07:08:29 pm
content: Quote from: QinX on March 13, 2015, 06:57:29 pmThere are 4 red PCIE slots (guessing for the hardly ever works quad SLI) and 8 multiplexers...

author: QinX
author_rank: Contributor
timestamp: March 13, 2015, 07:19:02 pm
content: Quote from: Mechanical Menace on March 13, 2015, 07:08:29 pmIn my edit I found out that each ASM1480 switches 2 full PCIe lanes, (2 RX pairs and 2 TX pairs, or however the manufacturer implements it) so to switch 8 PCIe lanes you need 4 ASM 1480.BUT, because these are used to allow for either 1 16x lane or 2 8x lanes they are usefull for motherboard manufacturers, but my design doesn't need that, so I shouldn't need these multiplexers. I just want to split the route the 16 lanes to 8 lanes. They question is, will motherboards/BIOS accept it.The challenge would be the none PCIe Data lane signalshttp://pinouts.ru/Slots/pci_express_pinout.shtmlI need to figure out what these signals are and what to do with them.1) SMBus Clock2) SMBus Data3) TCK4) TDI5) TDO6) TMS7) +TRST#8 ) Link Reactication9) Power Good10 and 11) Reference Clock pair

author: Monkeh
author_rank: Super Contributor
timestamp: March 13, 2015, 07:38:23 pm
content: You can't just split the lanes without appropriate hardware support.The magic search term you need is PCIe switch. The 32+ lane ones are, as you've noticed, quite expensive.

author: janoc
author_rank: Super Contributor
timestamp: March 13, 2015, 07:48:04 pm
content: Quote from: QinX on March 13, 2015, 07:19:02 pmI2C-like configuration and monitoring bus, just put the two connectors in parallel and it will likely work if the cards use it.Quote from: QinX on March 13, 2015, 07:19:02 pmJTAG. Useful if you want to debug the connected hardware, but then you must know how to access the JTAG chain on your mobo. Not likely to be very useful unless you are the manufacturer of the board. The cards probably don't really need this neither for normal operation.Quote from: QinX on March 13, 2015, 07:19:02 pmNo clue, should be described in the PCIe spec.Quote from: QinX on March 13, 2015, 07:19:02 pmLikely the system PSU power good signal. Used to tell the peripherals that it is safe to come out of reset and start powering up, because the PSU is stable already. Simply connect it to both slots.Quote from: QinX on March 13, 2015, 07:19:02 pmSystem clock distribution? No idea. Should be in the spec.

author: QinX
author_rank: Contributor
timestamp: March 13, 2015, 08:10:37 pm
content: Quote from: Monkeh on March 13, 2015, 07:38:23 pmMaybe, it might be simple in terms of hardware, hardware is just hardware and can be done by third parties. I fear it also needs software. ie the BIOS needs to do a handshake and have the code to allow for multiple cards. Seeing as a normal ATX/mATX have dual PCIe 8x slots and a mITX board with only 1 16x slot it should be doable from hardware, but if they removed the code in the BIOS to allow for this then it is going to be difficult/impossible.Quote from: janoc on March 13, 2015, 07:48:04 pmThank you for identifying 1 through 7.9 should have been obvious the first time I read it8 is a WAKE signal, it communicates the power state of the card to the host. It might not be needed but power consumption might become worse because it. Will have to look into it some ore.https://www.pcisig.com/developers/main/training_materials/get_document?doc_id=3ba5129826fa0214e5cd85fb70ff38e1df32a11c10 and 11)this is the 100MHz reference clock. This might be a tricky one.http://electronics.stackexchange.com/questions/93264/pci-express-bifurcation-clock-fanout-buffer-needed-to-split-reference-clockBut again if it is generated from 1 source and is supplied to multiple PCIe slots on some boards, there may be a simple answer.

author: Monkeh
author_rank: Super Contributor
timestamp: March 13, 2015, 08:15:53 pm
content: Quote from: QinX on March 13, 2015, 08:10:37 pmJust because the chipset can be used that way doesn't mean you can just split the lanes on any old board and expect it to work. Both the board and the BIOS need to be built to allow it.If you want to use an existing board you will need to use a packet switch. They are expensive if you want 32+ lanes.If you can get away with 12 lanes (x4 host > two x4 cards) they're more affordable.

author: QinX
author_rank: Contributor
timestamp: March 13, 2015, 08:31:15 pm
content: Quote from: Monkeh on March 13, 2015, 08:15:53 pmIt shouldn't need a 80 dollar chip to do it.Why? because of the following.Scenario:Chipset is not important for me, because Intel CPUs of the Ivy-bridge generation and later have >16 PCIe 3.0 lanes available directly from the CPU, no chipset is between the GPU and the CPU.The ASUS Maximus VII Hero has 2 PCIe 16x slots, but when 2 cards are installed these operate at 8x they only use the ASM1480 chips to change lane allocation from 16x/0x to 8x/8x no fancy chip needed.the ASUS Rampage VII Impact is based on the exact same platform as the Hero, but it is in mITX format so it only has 1 physical PCIe slot.What is stopping me from just adding the same components ASUS added to the motherboard to allow for the lanes to split?The only thing I can't do is change the code for the BIOS. IF it is needed, I suspect it will need it.If I want to be able to do it with every board, then I would need the PLX PEX8747. But seeing as that is no possible for me I want to figure out if I can do it the cheap way. If it turns out that you have to have a BIOS that supports it than I will know it can't be done without manufacturer cooperation.

author: Monkeh
author_rank: Super Contributor
timestamp: March 13, 2015, 08:36:31 pm
content: Quote from: QinX on March 13, 2015, 08:31:15 pmChipset, CPU integrated host.. same difference. It is the upstream, I don't care where it comes from.QuoteAnd it has presence pins, clock buffers, and logic to detect cards inserted in the extra slot and have the BIOS configure the PCI-E controller to operate the bus as two x8 ports.QuoteThe lack of all the above.

author: QinX
author_rank: Contributor
timestamp: March 13, 2015, 08:54:22 pm
content: Okay, I'm starting to understand, still not totaly sure though.How would this card work then?RSC-R2UG-2e4e8http://www.acmemicro.com/Product/13545/Supermicro-RSC-R2UG-2E4E8-LHS-Passive-2U-PCI-E-Riser-Card?c_id=356It has the following specsOutput Type: Output: (3) PCI-E x16 , Signal: (1) PCI-E x8, (2) PCI-E x4GenN: gen2: Yes , gen3: NoGPU / PHI Support:Auto Detect: YesCompatible System: 2026GT-TRF, 2026GT-TRF-FM407, 2026GT-TRF-FM409, 2026GT-TFThere are 5 undefined pins on a 16x PCIe slot, they might have used those, but would that be enough to add 2 more PCIe slots?

author: TheUnnamedNewbie
author_rank: Super Contributor
timestamp: March 14, 2015, 03:58:04 pm
content: Quote from: QinX on March 13, 2015, 08:54:22 pmMy guess this riserboard only works on certain server boards that supermicro makes (first guess, didn't actually do any research). Will prolly be cheaper to just supply some support for this on their mobos in case the user wants to use it. So the chips it lacks compared to that other splitter card will simply be present on the motherboard.

author: QinX
author_rank: Contributor
timestamp: March 16, 2015, 09:36:33 am
content: Quote from: TheUnnamedNewbie on March 14, 2015, 03:58:04 pmIt does only work on certain supermicro boards, but lets try and figure out why. If they adhere to the PCIe specification than they can't send any extra clocks or signals over the lanes.If they deviated from the spec the question becomes how much?There are 6 Reserved pins on a PCIe 16x slot. The could use those without changing the function of the allocated pins. Would that allow them to go from 1 to 3 PCIe devices?the first device receives the extra signals from the PCIe slots regular pins, the other 2 would have 3 pins each.I suppose they would only need the WAKE signal and perhaps the reference clock. which would be 3 pins total for each device,There are also a lot of GND connections, if they use those from some extra signaling then they are no longer adhering to the PCIe spec. Which should be fine for them if they only support certain boards of their own.I've shot SuperMicro an email to see if they would send me some technical information.

author: QinX
author_rank: Contributor
timestamp: March 26, 2015, 10:37:37 am
content: I've been in contact with SuperMicro, they tried to help but also didn't.In the end I order a Riser Card and worked out the traces.So only the Reference Clock seems to be needed as an extra, besides some BIOS changes I'd imagine.Hardware wise it doesn't seem as hard as it could have been, I just need to figure out a way to turn 1 Reference clock pair into 2 on the PCB.I found this, but I'm not sure it is what I need.IDT 9DB23 (Two Output Differntial Buffer for PCIe Gen3)If somebody can provide some insight?Edit:So I read the IDT 9DB233 datasheet, I think this is what I need.it's a 20-pin TSSOP, it has an SMBus for controlling the PLL, but I don't think it is needed. If you just keep all of it low then it should just perform as a fanout for turning 1 diff pair into 2.

author: QinX
author_rank: Contributor
timestamp: March 28, 2015, 07:54:37 am
content: Bump!I had email ASUS about the possibility of doing this.They also mentioned that although PCIe Bifurcation is support on current generation Intel Core processor it needs to have some code written in the BIOS for detection and setting it up.The PCIe Bifurcation is used on a lot of highend motherboard, but I want to use a M-ITX board, and guess what, only 1 PCIe slot means they don't need the code. So I'm going to have to persuade someone to add this codeI do believe that I am on the right track.

author: janoc
author_rank: Super Contributor
timestamp: April 02, 2015, 09:09:52 pm
content: If you actually succeed at making ASUS to write you a BIOS patch for a one off board, hats off ...Unless you are golf mates with the CEO or something at a similar level, it is about as likely to happen as hell freezing over - it is a fairly significant engineering work you are most likely not ready to pay for and not economical for them unless you are willing to buy at least few hundreds/thousands of those boards.

author: BlueBill
author_rank: Regular Contributor
timestamp: April 03, 2015, 04:11:21 pm
content: It'll never happen, this project was dead from conception.I'm curious what the OP was working on, but we'll probably never know.

author: QinX
author_rank: Contributor
timestamp: April 05, 2015, 11:11:29 pm
content: Quote from: janoc on April 02, 2015, 09:09:52 pmI think you overestimate changing a BIOS, you see special BIOS for overclockers all the time, so why couldn't it be done for something like this, most manufacturers should already have the code, it's almost just a matter of implementing it for a specific board.Now I'm not saying it is cheap and they will do it for free. But I also don't think they completly ignore it.Unfortunatly you probably need to have some marketing related friends at said manufacturers to get inside. no way a simple email would do the trick.Quote from: BlueBill on April 03, 2015, 04:11:21 pmNever say never, I believe that if the hardware is capable of it, then a software solution can be achieved, in addition to the extra PCB it would need.I want to run 2 GPUs on a M-ITX motherboard for a product I am working on.

author: BlueBill
author_rank: Regular Contributor
timestamp: April 05, 2015, 11:58:59 pm
content: So why does it have to be Mini-ITX, a pair of video cards if powerful are going to require a fair amount of space and cooling. Seems a forum is looking for the same thing, but to no avail. It's simply too small a market and cooling would likely be an issue.http://www.overclock.net/t/1515838/mini-dtx-hype-thread-itx-with-two-pcie-slotsIt might help if you could explain what you plan to do with dual GPUs? There might be other solutions, would a dual GPU video card work like the ASUS ROG MARS 760?

author: janoc
author_rank: Super Contributor
timestamp: April 06, 2015, 12:57:09 am
content: Quote from: QinX on April 05, 2015, 11:11:29 pmThe big difference is that they produce and sell tens of thousands of those enthusiast boards. So the extra work required gets amortized in the quantity manufactured. I don't think anyone will be willing to do such mod for a one off, especially for a board that wasn't designed for it so it isn't a matter of flipping some flag somewhere. It is just too much hassle and risk. Writing the code, QA, support, you have mobilized a fairly large team of people and resources that have to be paid - all that because of a single board. The cost of that simply wouldn't be economical and it is a lot cheaper for the OP to find another solution for their need.

author: BlueBill
author_rank: Regular Contributor
timestamp: April 06, 2015, 02:09:24 am
content: There's an ASRock H81M-DG4 which has two PCI slots.

author: Billy Bones
author_rank: Newbie
timestamp: April 04, 2016, 03:30:29 pm
content: Hello everyone,sorry to "necro" this old post, but I'm interested in the very same thing QinX is: two GPU PCIe cards into a single PCIe slot on a mini-ITX motherboard.QinX, did you abandon this idea in the end?Did you try out the SuperMicro RSC-R2UG-A2E16-A (http://www.supermicro.com/a_images/products/Accessories/RSC-R2UG-A2E16-A.jpg) or was it assuredly only compatible with some of SuperMicro's own server hardware?Has anyone already emailed SuperMicro about that?Is there any news about that BIOS code? Did you correspond with ASUS or someone else any further?I found your build log on Overclock.net, QinX and it's very interesting. Nice work! You have quite the footsteps to follow.Best regards,Billy

author: BurtyB
author_rank: Regular Contributor
timestamp: April 06, 2016, 12:20:10 pm
content: It depends what your requirements are but I've successfully used a load ofhttp://amfeltec.com/products/flexible-x1-pci-express-3-way-splitter/to connect 3 PCIe graphics cards (with x16 to x1 adapters) to a mini-itx MB with a single x16 port which worked OK (for use with opencl with minimal communication going on).

author: genBTC
author_rank: Newbie
timestamp: October 30, 2016, 08:30:57 am
content: Quote from: BurtyB on April 06, 2016, 12:20:10 pm"for use with opencl with minimal communication going on" <- sure x1 is fine for this.But for science, why not find out how fast she can go!There is now an additional product from the same company:http://amfeltec.com/products/flexible-x4-pci-express-4-way-splitter-gpu-oriented/Which uses an x4 host card branching out to a total of 4 physical x16 slot adapters (included). HOWEVER these seem to be attached by the same electrically x1 flat flex ribbon cables just like the other product. I was unable to find out which chip they were using; pic is not high res enough and its not itemized in the datasheet, but it is for sure a PLX chip.(Large ,has a heatsink shown elsewhere, and datasheet/specs-page seems to indicate that each of the 4 are all addressable simultaneously and without a bottleneck).It would be interesting to know how much these solutions cost, and if indeed it would be possible to design a better one, either with an x16 host card branching into 4 x4's, or the original intention of x16 into x8+x8. Either using a faster more expensive PLX chip to copy this solution, or with the quad ASMedia 1480 chips (how i found this thread). I have confidence in the BIOS bifurcation thing being possible.I see no reason for this hardware build to be impossible, just a matter of why.-Abei Villafane

author: davidm71
author_rank: Newbie
timestamp: January 07, 2018, 01:31:24 am
content: Hi,I successfully modded my bios on an Msi x99a motherboard and 6850k on the last pcie x8 slot into two using a Supermicro 8x to 4x4x card. This lets me use my onboard M.2 drive at 4x and another nvme pcie drive on the last slot also at 4x! Only issue is sometimes randomly after a reboot the link speed of one of the connected devices drops to 2.5 gt/s from 8 gt/s. Looking for possible solution if there is one?Thanks

author: Dennis_G
author_rank: Newbie
timestamp: January 20, 2025, 01:23:56 pm
content: Hi folks,Any chance that anyone has this datasheet for the ASM1480 laying around? The link below expired, they (ASM) do not want to talk to the average Joe and I am thinking to onboard on a journey that uses this IC duplicating a board that already uses it, just that I am missing some connections.Quote from: QinX on March 13, 2015, 10:06:30 amThanks,Dennis