Deadbugging PCIE 3.0 GPU 8GHz - Page 1
Forum section: electronics / projects

author: invalid_syntax
author_rank: Newbie
timestamp: March 13, 2024, 03:50:16 am
content: First time poster, if wrong board, please recommend (RF ?)I am trying to see the possibility of deadbug style connection to this GPU.https://i.ebayimg.com/images/g/p3IAAOSwz2plQqT9/s-l1600.png- Pins are 1.67mm aparts. Soldering doable (* would help is thin wire can be used)- Total of 32 Differential Pairs ( Or 16 if gpu supports x8 downgrade).- Connecting to a PCIE Male connector going to motherboard.- Using 3M Twinax for the cable -https://www.digikey.com/en/products/detail/3m/SL8802-08-20DN5-00/3837412- Wanted to test it before/while working on the PCB with proper socket.While i wait for the gpu to arrive or begin testing. I have following two major concerns.1. Instead of connecting the twinax directly to the pins, is it possible to use a twisted pair, or twisted pair + several gnds thin wire like magnet wire 1-2 CM to a landing pad with many pins from where the Twinax will continue.2. The twinax does not have a drain wire per differential pair. It has two GND wires at opposite ends of 4 diff pairs. Waiting for the cable to arrive as well. If the shield is not grounded, do i need to ground it or not necessary for diff pair as long as return line is present ?Edit for additional details.- Data sheet for cable:https://multimedia.3m.com/mws/media/673519O/3mtm-twin-axial-cable-sl8800-series.pdf- GPU is SXM2 P100

author: inse
author_rank: Super Contributor
timestamp: March 13, 2024, 03:28:04 pm
content: I have no specific knowledge about PCIe but I expect signal skew to be absolutely relevant here.The proposed cable has solid conductors, they would break off already during soldering.Good luck on that…

author: ajb
author_rank: Super Contributor
timestamp: March 13, 2024, 08:45:51 pm
content: Seems like it would be a LOT less work to design a simple adapter PCB, on top of being more useful. There's a lot that can go wrong here, not just with signal connections but you also need to somehow get 300W into the thing. Hard enough to get all of that right on a PCB, let alone hand soldering flying leads onto a 1.6mm pitch grid.Do you have experience designing things like this?

author: tooki
author_rank: Super Contributor
timestamp: March 13, 2024, 10:19:45 pm
content: Quote from: invalid_syntax on March 13, 2024, 03:50:16 amThis seems like a fool’s errand to me that will just result in a dead GPU and a bunch of wasted cable and time.Either design your PCB now, or contact this dude who is willing to share his design files for the same thing:https://www.reddit.com/r/nvidia/comments/l7gon4/is_it_possible_to_convert_from_the_sxm2_interface/

author: nctnico
author_rank: Super Contributor
timestamp: March 13, 2024, 11:28:43 pm
content: I'd design a test PCB for sure. Using wires is going to end up nowhere; PCIe (and similar high speed signals) really need tight impedance control.

author: langwadt
author_rank: Super Contributor
timestamp: March 13, 2024, 11:50:30 pm
content: Quote from: nctnico on March 13, 2024, 11:28:43 pmhttps://hackaday.com/2023/11/08/getting-pcie-working-on-the-new-pi-5/

author: langwadt
author_rank: Super Contributor
timestamp: March 13, 2024, 11:56:14 pm
content: Quote from: inse on March 13, 2024, 03:28:04 pmeach lane of PCIe is separate, afair the allowed skew between lanes for PCIe 3.0 is 6ns, that is a lot of wire

author: ejeffrey
author_rank: Super Contributor
timestamp: March 14, 2024, 02:29:03 am
content: Quote from: nctnico on March 13, 2024, 11:28:43 pmThat's not the problem really. High quality twinax is plenty good for pcie. Actually implementing it is another thing. Soldering 32 twinax pairs (16 each direction) directly to pins sounds like a nightmare and likely to mess up and destroy it, but the biggest problem is power and bypassing. Power planes in a PCB provide a low impedance path to the regulators and bypass capacitors. That's going to be basically impossible to do with dead bug construction.

author: invalid_syntax
author_rank: Newbie
timestamp: March 14, 2024, 03:06:04 am
content: > .. slow 8 year old Volta GPU with only 16GB of RAMMy though process was to test with a $40 card with made up heat shink just to get connections validated. After that was planning to move to P100, still an old card but does job running quantized models or small fine tunings.> The cost of spinning a board with an SXM2 interface conversion function is likely to massively exceed the cost of buying a modern GPU with 2-10x the performance of this card.At worst, will write it off as learning experience.Based on research so far (need to compile them and share it as well).- SXM2 differential Pins are 1:1 with PCIE Lanes. 32 Diff pairs as well as SMBUS, REFCLK and other pcie houskeeping pins.- Rest of the pins are power pins and NVLINK pins. NVLINK is optional can be left unconnected may be ?.- For LLM inference, major bottleneck is the GPU memory bandwidth. Can take hit in CPU to GPU bandwidth, once models are loaded, the pcie link is used less. So link downgrade might be acceptible.For PCB.- Aiming as frugal overall solution as possible.- The PCB will be bare bone SXM socket pins to PCIE male pins only.- In addition to data traces, For power, either PCIE GPU Header or plain 12V pins.- Looking at some of the PCIE extension cable's dimensions, 100x100 board should do it. PCBWAY or JLPCB's cheap 4 Layer should cover it.- Biggest hurdle/cost sink is the BGA socket with 400 pins. Out of two, only one is required if not using NVLINK.- My first experiment is going to be checking whether a PCB cutout can be used in place of socket. Basically a BGA pad placed on top of supporting structure that has screws to mount. And either soldering pins or high density headers. This is need to research more.

author: invalid_syntax
author_rank: Newbie
timestamp: March 14, 2024, 03:38:48 am
content: Quote from: ejeffrey on March 14, 2024, 02:29:03 amLooking at the on-board power controller. I was hoping, pins would be crude power source which gets properly distributed by onboard power controller.This is the reference design i am following.http://files.opencompute.org/oc/public.php?service=files&t=1b45195dfc4f7a8f204a246fb41f77b9I think cadence files. I dont have access to it, so using gerbers and kicad. Files are large, takes long time to load.

author: Miyuki
author_rank: Frequent Contributor
timestamp: April 21, 2024, 12:15:29 pm
content: PCIe can work just fine over this crazy cable, probably the same as was used for IDE drives. Chinese shops are full of those and surprisingly they work just fine.

author: amyk
author_rank: Super Contributor
timestamp: April 21, 2024, 07:54:35 pm
content: I suspect those ribbon cables are just barely adequate, but shielded ones perform much better. Here's an interesting video showing just how tolerant of delays PCIe is: