How to test salvageable Xilinx UltraScale+ board from Ebay? - Page 1
Forum section: electronics / repair

author: lukego
author_rank: Frequent Contributor
timestamp: February 05, 2020, 08:00:03 am
content: I'm a total newbie to electronics and I've made the rash decision to order a few boards from Ebay that include supposed-to-be-working Xilinx XCKU15P FPGAs. This is a $3500 part being sold for less than 1/10th of list price. I couldn't resist this because 100G ethernet is an important part of my domain of interest and this FPGA seems to support up to 8x100GbE links.Here's the product on Ebay:https://www.ebay.com/itm/Xilinx-Kintex-XCKU15P-Fpga-On-Board/274145389372The cards will arrive soon and I'm now wondering how to test that the FPGAs work (and otherwise return them.) Any ideas? How do you usually make sure the salvage parts you buy from Ebay are in working order? I'm planning to desolder these and put them onto custom boards but that will be well after the 30-day returns window and presumably desoldering them would void my quasi-warranty anyway :-)These boards are supposed to be (damaged in some unspecified way) Mellanox Innova-2 Flex MNV303212A-ADLT cards. If I'm lucky I will be able to put them into PCIe slots on a Linux machine and probe the FPGA indirectly using Mellanox software. However I don't know if the board and/or ConnectX chips are in working order and so maybe I will need a more direct approach. I don't currently have Vivado or a JTAG programmer but I should be able to arrange access to these if needed.Relatedly, what would be some good questions to ask the seller regarding testing?Thanks in advance for any sage advice for this rash newbie...

author: Haenk
author_rank: Super Contributor
timestamp: February 05, 2020, 08:12:26 am
content: A proper cable does not cost a lot - so if you install two cards in two computers, you should have a working connection - if the FPGA is OK. (Note that the FPGA probably does come with a huge heatsink, so you would need appropriate cooling while testing.)Another thing to note is, that those 100GbE-Adapters seem to have dropped in price a lot, there are plenty of those cards for far less than 350 USD around, however with the heatsink still installed, so one cannot be sure if this XILINX is on board (need to research that...)

author: lukego
author_rank: Frequent Contributor
timestamp: February 05, 2020, 08:25:52 am
content: Quote from: Haenk on February 05, 2020, 08:12:26 amDo you mean a cable for 100G ethernet? I do have that, and some other 25G/100G ethernet adapters, so I could test that way. But this would seem to depend on the board as a whole working which is in doubt.QuoteI should clarify: On the one hand it's true that 100G ethernet adapters are a relatively cheap commodity these days I don't think the same is true for FPGAs with SERDES capable of 25G/100G ethernet. It's the 25G/100G FPGA development platform that I want. So in a sense that I'm doing here is buying 32 x 32.75Gbps UltraScale+ GTY SERDES transceivers (for a song if they should happen to really work.)

author: brabus
author_rank: Frequent Contributor
timestamp: February 05, 2020, 08:41:27 am
content: Quote from: lukego on February 05, 2020, 08:00:03 amDon't want to be rude, but those beasts are an authentic nightmare to solder even with specific tools (vapour phase soldering is a must). Any "home made" trial WILL end up damaging the FPGA and wasting 350$.I would skip this phase and work on the development board until the VHDL script is satisfyingly robust.Once the prototype works on the DevBoard, you can design the application PCB and have a brand new FPGA soldered through an EMS service. This is pretty much the only way to have it working at the end of the process.If the seller says "used", it means that the boards still work, otherwise they shall be listed as "for parts/not working".

author: lukego
author_rank: Frequent Contributor
timestamp: February 05, 2020, 08:52:03 am
content: QuoteI'm planning to either resolder these FPGAs using hot air, or otherwise to dead bug them. I'll make a video to share my misadventures here so that others can learn from my follyMaybe I'll start carving notches on my hot air gun handle for each thousand dollar chip I bugger up while trying to imitate Louis Rossmann...

author: Fred27
author_rank: Unknown
timestamp: February 05, 2020, 09:01:01 am
content: Quote from: lukego on February 05, 2020, 08:52:03 amIf you're an electronics newbie than I'm afraid you may have jumped into the deep end (with some sharks) for your first swim.

author: OwO
author_rank: Super Contributor
timestamp: February 05, 2020, 09:13:45 am
content: Find the JTAG pins and attach the Xilinx programmer, then in Vivado you should be able to read device type and device DNA, which you can look up on Xilinx's website to check part number and date code. It does not guarantee that the entire chip is good, but at least can tell you it's not a dud. Next thing I would do is create a very basic project in Vivado and try to program it.For desoldering and re-soldering you need a hot plate/preheater and a hot air station. Remove the nozzle from the hot air gun for the widest air beam, set wind to maximum, and continuously circle the chip with the hot air gun. Also apply flux under the chip before desoldering.After desoldering clean the chip with a solder wick, but DON'T use a soldering iron. Set the hot air just hot enough to melt the solder blowing on the chip, add flux, and insert the wick slowly.

author: lukego
author_rank: Frequent Contributor
timestamp: February 05, 2020, 09:23:59 am
content: Quote from: Fred27 on February 05, 2020, 09:01:01 amSharks shmarks. Just punch 'em on the nose, right?!

author: Fred27
author_rank: Unknown
timestamp: February 05, 2020, 09:31:18 am
content: I like your attitude and I'm looking forward to the video. Good luck!

author: Haenk
author_rank: Super Contributor
timestamp: February 05, 2020, 09:33:13 am
content: (I don't think a DIY desolder, reball, solder will work - unless you own a ZEVAC or the like...)Best is to ask a company which specializes into BGA rework - e.g. we have a company located near our office which does official repair works for quite a couple of known brand mobile phones and lots of other stuff. They *do* have the skilled people and the proper equipment for such a task. Might be a good idea to locate such a company anyway near you...

author: lukego
author_rank: Frequent Contributor
timestamp: February 05, 2020, 03:14:54 pm
content: Quote from: OwO on February 05, 2020, 09:13:45 amThanks for the advice. Currently I have a Linux box with a USB MPSSE cable but no Xilinx/JTAG programmer and no Vivado license. I'll have to see if there's some open source software for programming via JTAG using the MPSSE cable and otherwise acquire a dedicated programmer. I'll hopefully be able to use a cloud-based Vivado to create a bitstream remotely, or else get a 30-day trial of Vivado to run locally, or else shell out for a license (yikes.)Quote from: OwO on February 05, 2020, 09:13:45 amThanks for the tips! I have a preheater and a hot air station, and I have intently watched several YouTube videos of people using the technique you described on e.g. GPUs (though often wicking up solder using an iron directly, is that really so bad?)

author: lukego
author_rank: Frequent Contributor
timestamp: February 05, 2020, 03:16:47 pm
content: Quote from: Haenk on February 05, 2020, 09:33:13 amSpecifically what do you think will cause problems when reworking with hot air? Just overheat a part of the chip and damage it? How come it's a problem for the FPGA while YouTube is full of people doing it successfully with iPhones/iPads/Macbooks/etc?Quote from: Haenk on February 05, 2020, 09:33:13 amI ordered four of these chips. If I kill the first then I'll be more careful with the second

author: Haenk
author_rank: Super Contributor
timestamp: February 05, 2020, 03:35:20 pm
content: Quote from: lukego on February 05, 2020, 03:16:47 pma) they won't annouce a failureb) those chips are usually way smaller, so they are easier to get the hot air below the chipc) professional repair guys probably did this hundreds or thousands of times before - that makes a differenceI suggest to test your skills on some scrap boards, before killing off such an expensive chip(But it's your money, so do as you like, of course...)

author: lukego
author_rank: Frequent Contributor
timestamp: February 05, 2020, 03:40:44 pm
content: Quote from: Haenk on February 05, 2020, 03:35:20 pmIndeed. I have been practicing on scrap PCBs from e.g. old Macbook Air laptops. Just removing chips with the hot air rework station and putting them back on with a soldering iron.The next practice step will be with BGA. I don't know quite what yet. Maybe just find some working but expendable device with a BGA, remove the chip, reball it and put it back, see if it works. Buy a replacement chip and repeat until that works. This should be fun and inexpensive if done with a cheap chip.Just right now the urgent need is to test the UltraScale+ chips as well as possible during the period when I can still send them back if they are faulty. This is up-front due diligence. Then I am happy to wait and develop my skills further before I actually desolder them.

author: OwO
author_rank: Super Contributor
timestamp: February 05, 2020, 03:42:27 pm
content: The free version of Vivado won't be able to synthesize for the part, but it should still be able to detect it and show device id. Xilinx JTAG cable is about $20 on aliexpress.I don't know which videos you watched, but my experience is most videos about soldering on youtube are utter bullcrap and teach you the exactly wrong techniques. Louis Rossmann is actually one of the "better" examples (compared to everything else on youtube that is) even though that's still not how you are supposed to solder.Using an iron with a solder wick has a high chance of ripping pads, especially if you don't yet have a good feel of the temperature settings of your iron. Too low and the solder will solidify between the wick and the board, too high and you will displace pads. Using hot air allows you to brush the wick gently over the pads.How do you plan to reball it? Apply paste onto the chip with a stencil and reflow? A thick stencil will be needed to get enough solder on it. The other technique that I'm experimenting with is to just skip the reballing and solder it as a LGA (apply paste on the board with stencil, place chip, reflow). If you use a thin enough stencil and get just a tiny amount of solder on each pad, you can get away with pushing down on the chip during reflow which will ensure all pads make contact. I don't know how well this works with large chips though.

author: lukego
author_rank: Frequent Contributor
timestamp: February 05, 2020, 03:58:33 pm
content: Quote from: OwO on February 05, 2020, 03:42:27 pmMy understanding, not yet verified first hand, is that Xilinx provide an Amazon EC2 image with the complete unrestricted Vivado for $0.36 per hour of usage here:https://aws.amazon.com/marketplace/pp/B07WNR73X5?qid=1580369452876&sr=0-5&ref_=srh_res_product_titleQuoteJust googling it seems like I should be able to use the MPSSE cable with OpenOCD to program arbitrary JTAG devices. Yes? (EDIT: Provided I have already generated a bitstream with Vivado.)QuoteCurrently I like the look of this method:https://www.youtube.com/watch?v=DVTxHx0z-wo.That's wicking up solder with an iron, placing solder balls by hand, using tacky flux as an adhesive to keep them in place while reflowing them with hot air. Crazy shit but you only live once amirite.I have practiced this on an ECP5. I'm pretty sure that I killed that chip ($5) by applying too much hot air, not seeing that the reflow had already happened. I'll do a few more dry runs until the technique feels comfortable, and then try actually resoldering a few real chips until I succeed, and then dare to desolder the UltraScale+.QuoteStencil may well be a better option. I have some solder paste and some thin generic stencils but I haven't tried any yet. I'm told that excellent custom stencils can be had from Aisler.net for cheap and maybe I will take that route.I think the reason my first attempt at reballing went badly is that the solder balls don't contain flux and don't look the way I expect when they reflow, but it could also be that I just missed it. I ended up cranking the heat to 500C and nearly burning a hole in my ESD safety mat... making my newbie mistakes!

author: lukego
author_rank: Frequent Contributor
timestamp: February 05, 2020, 04:14:00 pm
content: Just to share my journey of learning to solder and rework, here's a picture of the 0.8mm ECP5 that I cleaned up with wick and iron, then dead-bugged on one pad with a wire pencil:I'm quite proud of that effort!I'm less proud of this mark on my ESD mat where I heated that same chip almost certainly to death while making my first attempt at reballing:Here is the wire connection up close. Looks to me like it worked - the insulation melted in the solder as it was supposed to - but I didn't choose the angle well for strength:Incidentally, although I'm learning a lot from YouTube, I'm learning much more from reading the EEVblog forum archive.

author: james_s
author_rank: Super Contributor
timestamp: February 05, 2020, 06:29:20 pm
content: There is no way you are going to dead bug such a large FPGA and be able to get enough decoupling for the power rails in there and enough wires hooked up to be useful. You would have to be an insane masochist to try removing it from the board, it's already professionally soldered onto. By far your easiest and most likely to succeed path is going to be using the board you have.How much experience do you have with VHDL or Verilog? If you are starting from zero then figure on investing several years before you have anything even approaching a need for such a large part.You say you are a "total noob" and yet you are trying to tackle the sort of project that if someone with a PhD in EE and 5-10 years of experience in hardware design succeeded it would impress me. You are drastically underestimating the difficulty of what you are trying to do.

author: lukego
author_rank: Frequent Contributor
timestamp: February 05, 2020, 08:30:46 pm
content: Quote from: james_s on February 05, 2020, 06:29:20 pmYou say that like it's a bad thing?I'm excited about having a high-end part even for simple designs. Take for example the fpga4fun ethernet packet generator (https://www.fpga4fun.com/10BASE-T3.html). That usually runs at 10MHz using a normal I/O and this makes it a throw-away toy. If you ran the same program on the XCKU15P's GTY SERDES then you would be outputting over one billion packets per second at precise intervals. That's far from a toy... if you asked a vendor like IXIA for a quote on a load generation device with that bandwidth they'd probably want a milllion dollars.So I'm excited about this hardware and all the amazing potential that it has. It's something that I could use to do stuff that I can't do at all today with off-the-shelf cards. But I'll pursue this project in parallel with other more modest ones, like programming an off-the-shelf ICE40 board and making a custom ECP5/10G board, and I'll take it at the pace that my embryonic skills allow.For now it will serve as an excellent vehicle for learning enough JTAG to check device identity and write a bit stream. Who knows, maybe that will even lead me to discovering faults and sending the hardware back for a refund!

author: james_s
author_rank: Super Contributor
timestamp: February 05, 2020, 09:46:46 pm
content: Underestimating the difficulty of a project by orders of magnitude is a bad thing unless your goal is to waste time and throw away money banging your head against the wall. Those of us who have been around here for a while have all seen a few overly ambitious beginners crash and burn and disappear.I got a deal on a relatively high end FPGA board back when I first started developing with them, it was still much smaller than the ones you got, small enough to be supported by the free version of the software.I eventually ended up selling it after finding I had no need for such a cavernous FPGA and the large size just meant it took much longer to compile for it than for smaller parts.Your biggest obstacle is going to be the fact that the free software does not support that part so you are dead in the water unless you have the kilobucks to purchase a Vivado license or manage to find a cracked copy.The board itself that it comes on looks potentially quite usable, if you desolder it from that you may as well toss it in the trash or make a keychain because it will never function again.

author: thinkfat
author_rank: Unknown
timestamp: February 05, 2020, 09:51:06 pm
content: Quote from: james_s on February 05, 2020, 06:29:20 pmI'd say, a PhD in EE doesn't qualify for that kind of shit5-10 years of experience - probably.Deadbugging an Ultrascale+ FPGA - that sounds like a recipe for disaster. There's people wetting their pants when they think of designing a board for those :-DCan't wait to watch!

author: MadTux
author_rank: Frequent Contributor
timestamp: February 05, 2020, 11:40:36 pm
content: Don't these things require a lot of different voltages for correct operation, like Vcore, Vio, Vmemory etc for correct operation, applied in the correct order? With Vocre something crazy as 50A at 0.8V or something when running at full load? No way this is gonna happen dead bug style with low enough ripple and without lots of big low-ESR ceramic caps directly beneath the chip.

author: thm_w
author_rank: Super Contributor
timestamp: February 06, 2020, 12:26:48 am
content: Quote from: lukego on February 05, 2020, 03:58:33 pmWhat is the hot air station you are using? It should never be necessary to set the temperature that high. I think max out should be around 400C or so.This could mean you either do not have enough preheat or the wattage of station is not high enough.The best 863 clone stations are good value, and not that expensive. ~$180 for 1200W. Or the genuine one is ~$300.For a FPGA of this size, I would speculate you need >700W but thats just a guess.

author: YetAnotherTechie
author_rank: Regular Contributor
timestamp: February 06, 2020, 02:04:42 am
content: If you want to learn the chips, use the boards as they are, they (are supposed to) work, and there would be no variables beyond your coding skills. Make your goal achievable by setting it to: being able to communicate using the existing 2(?) lines of GBe serdes that drive your interest and that are already there completely wired for you. Once you have that, and if your interest is still there, *research* how to connect the fpga to the real world, there is no point removing them from a working system if you don't have anadequate pcbto solder them to (i'm not talking about the soldering part). For someone skilled in high-speed digital AND low voltage/high current/low ripple power suply, the pcb design for that beast is on itself a work for several months. Skip the frustration of not being able to do the ethernet stuff because you fail at the pcb design stuff.

author: DaJMasta
author_rank: Super Contributor
timestamp: February 06, 2020, 02:28:26 am
content: As mentioned repeatedly, do not even attempt to dead bug prototype this chip.Even if you can work like a machine night and day to absolutely perfect results the very first time (and you cannot, make no mistake), all of your decoupling caps will be too far away to have any effect, all of your controlled impedance traces and differential pairs will be mismatches, and you will never see anything close to the full performance of the device if you can get it to boot at all. And this is if you can actually execute 1500 pins worth of fine pitch dead bug soldering without accidentally mashing them and ripping balls off the chip, coughing and getting dirt in a place that cooks into a short, or otherwise.With packages like this reflow is the only option. Vapor phase soldering may not be a requirement, but at the very least you need a board with a footprint, a preheater, a good hot air station, a BGA stencil appropriate for it, good solder paste, and good flux, ideally stuff designed for this kind of soldering. That's a bare minimum even with good soldering skills.I'm not the kind to tell people not to do something because they can't, but it is frankly foolish to attempt such a huge project without understanding the constraints. 1500 fine pitch contact points (a few hundred of which are probably mandatory just to power the device up) should dissuade most from attempting it... but the physical limitations of such a layout are too great to overcome with just that kind of persistence, unless your target clockrate is just a fraction of what it can operate at. The reason decoupling caps are put on the bottom of the PCB for big BGAs isn't just to save space, it's because the loop area at high frequencies to and from the cap can make it functionally nonexistent if it was put on the top of the board, next to the chip, in many cases.Anyways, see what you can do with the chips on the boards they are already on - it will be much easier with decoupling, power, and soldering taken care of, and then you can try to find/breakout IOs and things to reprogram at least somewhat to your desires. When you get some comfort with dealing with the chip and you have at least one that works, maybe try fabbing a carrier board for the chip in a CAD package. It will be an immense amount of work, dozens of hours, but with a board from a PCB fab with tight enough tolerances to accommodate such a big chip, you may actually be able to keep decoupling caps and rail generation close enough to use the chip normally and take advantage of all of its IOs. In the mean time, trying similar work with MUCH smaller chips would probably help develop the skills and understanding needed to deal with such a beast.To that point - maybe an early project would be to try and dead-bug some DDR memory. I'd be impressed if you could even get it to work (and that's not a slight on your skills, this comes down to layout restrictions and precision). Again with trace length matching and impedance control, even a chip with a tiny fraction of the number of balls and operating at a lower frequency, I don't think that memory would be usable at rated speed by a normal ARM MCU in a prototype situation without a PCB. There's a possibility, sure, and it may be a fun exercise to try..... but for the big timesink it will be, at least it won't cost hundreds of dollars and it should give you an idea of how difficult the task you're describing is.

author: OwO
author_rank: Super Contributor
timestamp: February 06, 2020, 03:49:40 am
content: Don't even think about deadbugging anything and yes, practice with smaller chips. I have a simple 5x5 cm Zynq SoM design on 4 layers that can be fabbed for $15:https://github.com/gabriel-tenma-white/sdr5#zynq_som_2The Zynq chip can be found for under $20. You can omit the DDR3 at first, test the FPGA functionality, and solder it in later. It's standard 96 pin DDR3, for example H5TQ4G63AFR.

author: lukego
author_rank: Frequent Contributor
timestamp: February 06, 2020, 07:47:38 am
content: Quote from: DaJMasta on February 06, 2020, 02:28:26 amConvinced on this point: I won't.QuoteI have gear recommended by Louis Rossmann and/or SDG Electronics YouTube channels. Quick 861DW hot air station, Amtech tacky flux and solder paste, a Yinhua preheater. I don't have good soldering skills yet, but I am willing to spend some weeks/months/years developing them for this project if necessary.QuoteI will be satisfied if the final result of this project is simply that I understand the constraints for the future i.e. education. ("Good judgment comes from experience. Experience comes from bad judgement.") Getting working boards is a "stretch goal." I'm also quite willing to find creative ways to cheat... for example I don't need many pins connected, only power and JTAG and one or more sets of TX/RX differential pairs for GTY, and so I might be able to make a custom stencil that leaves a lot of blank space and reduces the risk of shorts. Just having the chips there calling me to pursue these thought processes is really valuable to me. Maybe my learning style is unusual. I can't imagine anything more boring than sitting down for a few years to simulate all my designs in Verilator before melting some metal.It might bear repeating that these are not super expensive shiny new working boards that I'm talking about, they are salvage boards with unspecified defects bought for pennies on the dollar from Ebay. If these were new parts I don't think I'd even dare to take them out of their wrapping. They aren't though and that's why I'm using them for some high-risk high-reward soldering/electronics adventures.QuoteI will see what I can do. This sounds like a challenge though because this board does not connect the high-speed I/Os on the FPGA. Instead it connects the I/O up to the ConnectX ASIC and uses (wastes) the FPGA as generic compute offload.However, I have been sent some links to Aliexpress stores where people are producing custom boards for these high-end parts, and so if it turns out that I need a professionally designed board then one should be available. Likewise if it turns out beyond my capabilities to resolder the parts then i will look into relevant services, but that's not going to be Plan A.QuoteIndeed. The main thrust of my "learning hardware" project is to work with Lattice ECP5 FPGAs that are cheap and routinely used in projects by hobbyists. However, those are borderline useless for any real world applications in my domain of interest (telecom) and so I need to see some path to 25G/100G ethernet on the horizon to keep myself engaged and motivated.QuoteThanks for that tip. I do need to identify some suitable soldering/reworking/deadbugging practice projects for which I can identify whether the end result was successful or not. I do surely have a bunch of DDR chips on various old electronic devices like laptops and phones. I will be sure to start a thread if I attempt this.

author: lukego
author_rank: Frequent Contributor
timestamp: February 06, 2020, 08:01:02 am
content: Quote from: barycentric on February 06, 2020, 01:31:30 amGive me a break. This board is powered by PCIe.

author: lukego
author_rank: Frequent Contributor
timestamp: February 06, 2020, 08:20:45 am
content: I was privately recommended a book called "High-speed Digital Design: a handbook of black magic." I'll look into picking up a copy of that. There seem to be video lectures available too.

author: OwO
author_rank: Super Contributor
timestamp: February 06, 2020, 08:24:19 am
content: If you just want to play with gigabit transceivers there are Artix-7 XC7A100T boards for under $100. It's a huge step up to go from Lattice CPLDs right to ultrascale+ with 25Gb/s transceivers. The PCB design above 4GHz becomes difficult, so if you are designing your own boards I'd suggest to keep to Artix-7 speeds (6Gb/s) for now. You can always combine many lanes to get higher bit rates.

author: lukego
author_rank: Frequent Contributor
timestamp: February 06, 2020, 08:30:49 am
content: Quote from: OwO on February 06, 2020, 08:24:19 amThe minimum interesting speed for me is 10Gbps. I have already bought a Lattice ECP5-5G ($50) and an XGMII 10G PHY ($20) that I need to learn how to design a PCB for. This will be a challenge because all the SERDES on the FPGA will be dedicated to PCIe Gen2 x4 leaving only normal I/O for talking with the PHY. Hence the old-school XGMII connection that will be a challenge to route.The low-end Xilinx boards would probably be easier but I'm attracted to the Lattice because I can use a fully open-source toolchain (Yosys/Trellis) and then also learn about - potentially even contribute to - the software infrastructure like place&route. My background is developing software, mostly open source, and so it is with great reluctance that I use proprietary vendor tooling like Vivado. However I'm willing to make an exception if it opens the door to high-end I/O that I could use professionally.

author: OwO
author_rank: Super Contributor
timestamp: February 06, 2020, 09:02:18 am
content: 10Gbps = 5GHz. Probably doable on FR4 but you have to be careful about all discontinuities (e.g. via transitions) and route it as a microstrip pair with via fencing. For ultrascale parts the power distribution network is also difficult because you are dealing with tens of amps at 0.8V and it's sensitive to any voltage drop. I would personally use at least 6 layers for this, although smaller parts like XC7K160T might be doable on 4 layers. If you don't have prior experience with high frequency PCB design I would suggest attempting devices in this order: (1) Zynq or small Artix (XC7A100T) (2) Kintex (XC7K160T) (3) Ultrascale+.

author: DaJMasta
author_rank: Super Contributor
timestamp: February 06, 2020, 09:04:09 am
content: If you're interested in 10Gbit/s sorts of speeds, I'd suggest taking a look at radio electronics and high speed design. All sorts of physical constraints because of the increasing role of parasitics with higher frequencies, issues resulting from wavelengths, and the value of impedance matching. We like to think of digital electronics as straightforward ones and zeros, but at high speeds the best we can do is analog with sufficient margins to be read as digital, and it becomes much more of an exercise in maximizing overhead and budgeting for irregularities, losses, and the inherent limitations of precision in devices being used.Not quite the same factors as low speed design, and while at 10s of MHz you can get around some of them especially just by using modern, integrated parts, you get into the 100s of MHz range and you can't avoid the effects, and you get into the GHz range and without factoring in the high speed characteristics of the physical layout of the board along with the parts involved, it's not likely to do anything that you intend. While it can be complicated, but it is interesting, and if you're looking for high speed devices and connections, it will be very useful!

author: switch998
author_rank: Contributor
timestamp: February 06, 2020, 11:44:07 pm
content: Quote from: lukego on February 06, 2020, 07:47:38 amThis will be hell on earth, turn back now. Also, without a preheater, you are going nowhere.I am saying this as somebody who has attempted the exact same thing as you (with almost 10yr of experience). The requirement was to transfer two 672pin FPGAs onto new PCBs, for three prototype boards. We only had a handheld hot air station and preheater. After an entire week of leaving work at midnight, smelling like flux, I wound up with the three working boards as required. Most of the time was spent in prep; removing FPGAs, cleaning, reballing, cleaning the board, then reflowing and testing, only to find one or two balls had combined, or the FPGA was cactus... about 2hr per reflow. Sometimes the system would work, and we would get to the final stage of testing and find corrupted data. And when we did get the three protos done and delivered to the customer, the reworked FPGAs failed in the field after a few months of operation.The lesson we learned was our lab was not equipped to handle that job. We spent a large sum of money on a proper rework station shortly afterwards. If you are serious about this, you should be practicing removing, reballing and reflow soldering a GPU on one of your laptop boards.

author: thm_w
author_rank: Super Contributor
timestamp: February 07, 2020, 12:09:36 am
content: Quote from: switch998 on February 06, 2020, 11:44:07 pmIts in the part you quoted.

author: lukego
author_rank: Frequent Contributor
timestamp: February 07, 2020, 02:25:57 pm
content: Found an interesting document from Xilinx that covers this part:UltraScale Architecture PCB Design - User Guidehttps://www.xilinx.com/support/documentation/user_guides/ug583-ultrascale-pcb-design.pdfJust at first glance I'm extremely impressed with the quality of this documentation.Looking at the power section, it looks like there are multiple power supplies, but that it's also possible to run everything at the same voltage so that you only strictly need one (with caveats, I'm sure.) Looks like you need about a dozen decoupling capacitors and yes I suppose (as mentioned above) those need to be close on the PCB and can't be e.g. on a breadboard hooked up to a deadbugged UltraScale.I'll have a closer read when I have a chance but meanwhile I just wanted to share it. If the Xilinx documentation is all this good then maybe I'll stop being so provoked by their pricing on Vivado etc.

author: miken
author_rank: Regular Contributor
timestamp: February 08, 2020, 10:12:50 am
content: The Xilinx documentation is not bad, the problem is that you need to read All Of It. If you don't read enough then it's guaranteed you'll miss something and screw it up.The AWS Vivado is a good idea... just make sure it's not locked to the particular part they use.The Howard Johnson books are fairly out of date. They talk about DIPs on a manhattan-routed board. The rules of thumb roughly apply still, but if you're serious about this get something like Bogatin.Now assuming you're still crazy enough to go through with this... You need at least Vcore, Vaux, VMGTAVCC, VMGTAVTT. (Vcore and VMGTAVCC could be the same but I wouldn't.) They need to be sequenced in a particular order. Fast SERDES suck a lot of power so run your design through Vivado first to get power estimates.But really, start with a less complicated FPGA, honestly...

author: lukego
author_rank: Frequent Contributor
timestamp: February 08, 2020, 09:56:23 pm
content: Quote from: miken on February 08, 2020, 10:12:50 amThanks for your tips. I am starting with a less complicated FPGA, the Lattice ECP5, but for better or worse I will have this UltraScale+ on my shelf beckoning me to try learning how to put it to use... I'm not in any hurry.

author: lukego
author_rank: Frequent Contributor
timestamp: February 14, 2020, 04:47:17 pm
content: Quote from: brabus on February 05, 2020, 08:41:27 amThank you for mentioning vapor phase soldering. I had never heard of this before. I've read all about it in the forum archives now and I love the idea. I can see that a vapor phase soldering capability combined with a properly designed PCB might upgrade my project from "crazy long shot" to something that actually ought to work.Seems like I could buy my own vapor phase soldering setup for between $200 (DIY galden in a pot) and $2000 (Imdes Mini.) That's real money but it would be awesome if I could use it to extract value out of these kinds of cheap salvage chips. Just these four chips that are waiting in customs now would cost around $15,000 to buy new and at that price I'd never dare to take them out of their packaging.

author: lukego
author_rank: Frequent Contributor
timestamp: February 18, 2020, 01:59:35 pm
content: Here is a related project: a guy who found some cheap Xilinx Virtex FPGAs on Ebay and made a custom development board for them:https://andybrown.me.uk/2016/06/20/virtex-e/I'm not sure how the difficulty of that project relates to mine. He's obviously extremely skilled and experienced - no comparison to me - and he is working with older FPGAs. Nevertheless I find this encouraging.

author: OwO
author_rank: Super Contributor
timestamp: February 18, 2020, 02:50:26 pm
content: That Virtex E has as much logic capacity as a small spartan 6 and draws up to 1.5A at full load, and it's in a small 676-ball package, the same as the Artix board I'm working on.The Artix draws up to 6A and I'm using 4 layers, but the power distribution design is already difficult and I'd say this is borderline for a 4 layer board. Mainly it's the MGT region that is blocking routing because the transceiver pins can't be all escaped on the top layers, meaning you are using 2 signal layers + 2 ground planes in that area. Power planes there will *not* do because the MGT area requires regular stitching of all planes. The routing would be far easier if you simply do not use the MGT, but that defeats the point. Incidentally I looked at the QMTech Artix-7 boards and it seems they ignored the MGTs completelywhich is why I'm designing my own board.For larger Kintex I would say you *really* do need a 6 or 8 layer board if you want to actually use the transceivers (which btw you aren't going to route correctly if you aren't familiar with RF PCB design).

author: brabus
author_rank: Frequent Contributor
timestamp: February 18, 2020, 03:32:11 pm
content: Quote from: lukego on February 18, 2020, 01:59:35 pmLukego…J U S T D O I T .Don't forget to update us with pictures!

author: lukego
author_rank: Frequent Contributor
timestamp: February 18, 2020, 03:44:32 pm
content: Quote from: brabus on February 18, 2020, 03:32:11 pmRight on!I am going to. I reckon the first steps will be to wait until the chips get through customs (some weird holdup), then buy 250ml of Galden 230 and an asparagus pot, then desolder one of the chips by reflowing it in the galden vapor while gently pulling it with some kapton-taped wire.Then the PCB comes into play... and I will be looking for shortcuts/cheats e.g. premade boards on Aliexpress...

author: brabus
author_rank: Frequent Contributor
timestamp: February 18, 2020, 03:54:07 pm
content: Quote from: lukego on February 18, 2020, 03:44:32 pmThat's my boy!

author: lukego
author_rank: Frequent Contributor
timestamp: February 18, 2020, 06:00:53 pm
content: Customs say the chips are on their way now so I've ordered the Galden LS 230 and asparagus pot

author: lukego
author_rank: Frequent Contributor
timestamp: February 19, 2020, 06:09:50 pm
content: Bugger!I went to collect the chips/cards today and customs wanted $500 duties on a $1000 package. That's highway robbery. I'll have to call them up and see what the story is. Hopefully it's a mistake and not e.g. that Sweden has punitive tarrifs on imports from Israel. I've not heard of tarrifs above 25% before but I moved to Sweden recently and don't know all the gotchas.

author: OwO
author_rank: Super Contributor
timestamp: February 20, 2020, 03:43:40 am
content: So there is VAT and then there is import duty... both will be charged when importing from outside EU. VAT is 25% except for select few types of goods (food etc), and import duty varies depending on what HS code the sender declared:http://tulltaxan.tullverket.se/But it can be as high as 20% which means 45% total tax rate, which I guess is considered normal in Europe

author: lukego
author_rank: Frequent Contributor
timestamp: February 20, 2020, 04:59:27 am
content: Quote from: OwO on February 18, 2020, 02:50:26 pmHow about if I'd strip the board down to the barest of essentials to test 25G Ethernet: power, JTAG for programming, and two SERDES differential pairs for TX/RX. Is that starting to sound more feasible as a first step?Six-layer boards seem to be available for pretty reasonable prices from JLCPCB so that should be okay for an "advanced" stage prototype but quite slow and expensive if I'd try to start with "hello world" and iterate from there. Hopefully it's possible to do something involving 25G/100G on four layers as a warm-up exercise.

author: lukego
author_rank: Frequent Contributor
timestamp: February 20, 2020, 05:03:22 am
content: Quote from: OwO on February 20, 2020, 03:43:40 amThanks for the reference. I'll call them up and ask some newbie questions. Thankfully I incorporated a company as a vehicle for my hardware adventures and so it's exempt from most taxes and is usually able to reclaim VAT. I've never quite been sure how this applies to imports yet though, nor how much headache it can be when I forget to ask AliExpress shops to put the correct value on their declarations...

author: lukego
author_rank: Frequent Contributor
timestamp: February 20, 2020, 07:54:02 am
content: Ooops!The package only has my name on it, not the company name, and this means that it incurs punitive taxes that the company can't claim back later. So I have to return the package to Israel and ask them to take a pen and write the company name on the label and then send it again.Sweden... come on... you are better than this..

author: OwO
author_rank: Super Contributor
timestamp: February 20, 2020, 08:39:42 am
content: Quote from: lukego on February 20, 2020, 04:59:27 amLooking at the datasheet, the XCKU15P has a quiescent Vccint current of 2.7A, and the start-up current is Iccint + 3.5A, or 6.2A. All the other rails are <2A total. For comparison the PDN design on my Artix 4-layer board can handle 8A @ 0.03V drop, so yes it's probably possible to "power up" the XCKU15P on a 4 layer board. There's not going to be much margin for the user design though, and it's kind of a waste of an ultrascale+ FPGA. But yes it seems on a 6 layer board you can probably get a good enough PDN and route out enough user IO to get some basic use out of the FPGA. I haven't looked at the pin layout of the transceivers though so I can't say if it's possible to route out all the transceivers and maintain signal integrity.

author: OwO
author_rank: Super Contributor
timestamp: February 20, 2020, 08:50:54 am
content: Ultimately it will depend on your PCB design skills. It seems most people still need 8 layers to do a Zynq + DDR3 design and still struggle with signal integrity. If you contract a typical PCB designer and ask them to do Ultrascale+ and MGT on a 4/6 layer board they will laugh at you, but I think it's within the realm of possibility at least for 4-8 lanes of MGT (on 4 layers). The Artix board I'm working on already brings out 8 TX and 8 RX lanes. Maybe I can do the design work in exchange for a few FPGAs

author: thinkfat
author_rank: Unknown
timestamp: February 20, 2020, 09:14:29 am
content: Quote from: lukego on February 20, 2020, 07:54:02 amInteresting - so it's not enough that your VAT ID is on the invoice? Usually, customs inspect the invoice that goes with the shipment, not the address it was sent to.Or did you mean to say that you as a private person have to pay taxes that your company need not?

author: lukego
author_rank: Frequent Contributor
timestamp: February 20, 2020, 12:18:44 pm
content: Quote from: thinkfat on February 20, 2020, 09:14:29 amThis is the part I got wrong. The invoice did not show the company details. I thought they would just charge me domestic VAT on the import and give me a receipt for that so I could claim it back via the company, as with the VAT on stuff I buy locally. I was wrong. Live and learn...(I had thought this was drifting off-topic but oh boy is it important to understand tax/duties/VAT rules when you are setting up your electronics lab! Here it seems to be totally different rules for domestic verses EU verses non-EU.)

author: lukego
author_rank: Frequent Contributor
timestamp: February 20, 2020, 01:34:16 pm
content: Quote from: OwO on February 20, 2020, 08:50:54 amPardon my ignorance but to what extent can you "throw layers at the problem"? Is it easy to design a 12 or 24-layer UltraScale board? Is that one effective way for a low-skilled designer to cheat?

author: OwO
author_rank: Super Contributor
timestamp: February 20, 2020, 02:35:05 pm
content: Yes and no. Adding more layers will make it easier to have a solid ground plane against all signals, but it won't give you more interconnect density when it's via bottlenecked (vias go through all layers and annular rings must also be present on all layers). It makes BGA fanout easier (you have a new set of "channels" between vias on every layer), but doesn't help you with PDN design until you get to >8 layers. Inter-plane capacitance becomes usable when plane distance is 0.1mm or less. You still just have the same tiny amount of space to place all your decoupling caps. If you try to wing it with minimum effort (dedicated plane for all rails, textbook fanout strategy/auto-fanout, etc), you could end up throwing 14 layers at it to get everything routed, but an optimized design might use 6 or 8 layers.

author: OwO
author_rank: Super Contributor
timestamp: February 20, 2020, 02:45:09 pm
content: I re-read your question and the answer is, it's a way for experienced designers to cheat and take less time/effort, but no I don't think more layers is going to help you if you haven't done much PCB design and want fast & easy designing with signal integrity. I find that most of the work isn't routing but placement, and signal integrity isn't just about ground planes. All layers must be stitched near signal transitions, and this gets harder to do when you have power planes.

author: lukego
author_rank: Frequent Contributor
timestamp: February 20, 2020, 03:05:51 pm
content: Fascinating!So are there any other ways to cheat when you are prototyping? Can you stack several mezzanine boards together to make life easy? Somehow use USB3 cables to protect signals? Or does everything like that only make it harder?

author: OwO
author_rank: Super Contributor
timestamp: February 20, 2020, 03:39:36 pm
content: The hardest part is going to be the PDN, and there you *really* want to minimize distance as much as possible because you are transporting tens of amps of current while the FPGA only tolerates a Vccint drop of <0.03V. Routing difficulty is all going to be in the BGA area, so board stacking isn't going to help. The Vccint trace is going be very fat and be paralleled on several layers, so will use a big chunk of the area under the BGA where no other trace can be routed. You also need to position the vias so that the Vccint area is almost never interrupted by other vias.

author: OwO
author_rank: Super Contributor
timestamp: February 20, 2020, 03:53:57 pm
content: To get an idea here is the Artix board:The Vccint trace is duplicated on 2 layers. The MGT quad contains 8 lanes (4 tx + 4 rx) and takes all 4 layers (may not be interrupted by other routing). There is one more quad further down (not shown). On 6 layers I would dedicate a portion of one layer to Vccint (and maybe supplement it with fat traces on other layers), and you can also have rails go over the MGT and other I/O area with some care.

author: lukego
author_rank: Frequent Contributor
timestamp: February 20, 2020, 04:22:08 pm
content: Just wondering: is there any radical alternative to PCBs? Some other way to connect the BGA with power and IO?I ask because we're talking about this being very difficult and yet I have a bunch of sturdy cables that carry power and 100G Ethernet over long distances (e.g. a meter) without any fuss. So maybe PCB is the wrong idea since the goal here is not to connect the FPGA with other ICs on the same board anyway but just hook all of its IO up to external devices?Is there a way to do dead-bugging of the IOs that doesn't suck for example? Some miracle shielded wire?

author: OwO
author_rank: Super Contributor
timestamp: February 20, 2020, 04:28:56 pm
content: Do the math. You need to transport >20A of current with less than 0.03V of voltage drop, plus you need to have lots of decoupling caps less than 5mm from the pads. If something is hard to do on a PCB what makes you think deadbugging it will improve anything? I think you need to go and actually design a few FPGA boards (following all vendor layout recommendations), hold the boards in your hands and get an idea of the distances and dimensions. And think again if dead bugging or ANYTHING other than a PCB is really an option.

author: lukego
author_rank: Frequent Contributor
timestamp: February 20, 2020, 06:47:59 pm
content: Quote from: OwO on February 20, 2020, 04:28:56 pmI'm sure you this is true. Thank you for humouring me in this discussion. I recognize that I know "so little that I don't even know what I don't know" here.Having said that, here's one chip I have lying around that piques my curiosity, a Xeon E5:This comes in a package with the bypass caps built on and then about 1Tbps of IO (PCIe + QPI + DDR) mapped to big pads that don't need to be soldered but can attach to another PCB via a socket adapter.Couldn't one have such a thing for an FPGA that takes care of the power and then fans out the IO on a large package that's easier to deal with?(One other cheat I have in mind is that in Telco when Ericsson etc are testing radio equipment it's often far too difficult and expensive to actually use radio waves and they often use cables instead. Fine for prototyping and testing though hopeless for a production mobile phone!)

author: YetAnotherTechie
author_rank: Regular Contributor
timestamp: February 20, 2020, 07:54:28 pm
content: Quote from: lukego on February 20, 2020, 07:54:02 amQuick note:You can't simply "return" the item to Israel or whatever country it came from, and get rid of the obligation to pay import tax+duty.Your word "but i returned it through the post, here's the tracking number" means nothing.To "cancel" that bill you need to ship the item backtrough a customs officepreferably the one that did the importationusing their specific return/refund procedure. A local post office will not help you. If you ship it abroad and then import it again, with presumably the same invoice, one for final-user and another for a company, i wouldn't be surprised if you got some investigation about introducing the same item twice in the market, (that's how customs sees importing material), also, if you don't do the proper export/refund procedure, even if you don't pay tax on the second import, youwillpay import duty, duty is always due when first introducing an item to market, which if you don't cancel the first import is what your second item will be considered. Oh, by the way, the company that sold it to you will have to pay too when it enters in Israel...TLDR:Your item enters europe through a country's customs door, they attach duty/tax and send it to you.If you ship the item abroad trough another door, when it returns it will pass trough the same door above, where they attach the duty/tax again, ad infinitum.

author: lukego
author_rank: Frequent Contributor
timestamp: February 20, 2020, 08:06:02 pm
content: Quote from: YetAnotherTechie on February 20, 2020, 07:54:28 pmThis actually makes me feel better about Sweden :-). The customs people at the post company assured me that I won't be liable for any charges if I decline to accept the package and have it sent back instead. So I suppose they have a relatively integrated system in which the act of signing for a package from their agent completes its import into Sweden?(But I'm going to mail them one last time to check on this now that you mention it because your description sure does ring true.....)

author: YetAnotherTechie
author_rank: Regular Contributor
timestamp: February 20, 2020, 08:20:58 pm
content: I interpreted your message as "i picked the item and the bill", but if you haven't actually accepted it then yes, it should still be resolvable through the post office.

author: lukego
author_rank: Frequent Contributor
timestamp: February 21, 2020, 09:19:22 am
content: Here's my latest attempt to "think outside the box"Goal: Connect UltraScale+ chip with power and masses of high-speed (25Gbps) I/O for prototyping high-speed ethernet applications.Problem: Hard to design a PCB that routes I/O to the edges of the board with good signal integrity.Solution: Make a small PCB (approx same size as the FPGA) that drops all connections as directly as possible down to the bottom of the board using vias. Bottom of the PCB will look similar to the Xeon package above: decoupling capacitors for power in the middle and then I/O pads/connectors towards the edges that will be directly hooked up to suitable cables e.g. high-speed twinax likehttps://www.mouser.se/ProductDetail/Molex/100144-6800.Then (problem #2) somehow connect those twinax wires to 25G/100G ethernet SFP28/QSFP28 connectors.

author: OwO
author_rank: Super Contributor
timestamp: February 21, 2020, 09:39:29 am
content: No, that's not the problem. The problem is MGT area requires all 4 layers and "blocks" the area, meaning you can't place the PDN there or put decoupling caps. Trying to work around that with micro-twinax is going to give you far worse signal integrity than simply routing the PCB ignoring signal integrity, and power supply wires more than a cm or two is going to have more resistance than a suitably wide trace on the board. I think your best options are: (1) 4 layer board, Iccint supported up to 15A or (2) 6 layer board, Iccint supported up to 30A. Routing out the MGT pins is trivial, the issue is simply the density of vias there create holes in the power planes and limits max current.

author: thinkfat
author_rank: Unknown
timestamp: February 21, 2020, 09:42:42 am
content: That won't achieve anything. You'll have to fan out the BGA anyway and it will not make the task of designing the power distribution any easier. Effectively, you'll have doubled your problems. When you're done with the "adapter" board, you'll realize that you should have made the "adapter" bigger to make all the support components fit. And "bigger" means, you should have made the actual PCB in the first place.

author: OwO
author_rank: Super Contributor
timestamp: February 21, 2020, 10:08:10 am
content: Here is the pin diagram of the XCKU15P in FFVE1517, from document UG575.I've circled a few MGT quads at random. The cluster of pins at the top and bottom sides are general IO pins.Unfortunately, with this MGT pinout I can route at most half of the tx/rx pairs in each quad on a 4L board, unless you omit a few quads.The PDN specification will completely depend on how many ordinary IOs (not MGT) you want to break out. For example, if you break out all possible MGTs but zero IOs, you can see from the pin diagram that the maximum possible Vccint uninterrupted trace width is 20mm. This is going to be somewhat broken by GND pins, so let's say 15mm. The trace will exit from both upper and lower sides in parallel, so double that for 30mm effective width. Using a trace resistance calculator, assuming 30mm trace distance in bottleneck area, gives us 0.000590 ohms. 0.03/0.000590 = 50A. So far so good, but every I/O you add will create a hole in the PDN.So the real question is, how many MGT pairs do you want and how many general IOs?

author: lukego
author_rank: Frequent Contributor
timestamp: February 21, 2020, 03:36:15 pm
content: Thanks again for all the patient explanations. It sounds like it's time for me to try some reading and thinking "inside the box."

author: Haenk
author_rank: Super Contributor
timestamp: February 23, 2020, 01:49:12 pm
content: Quote from: lukego on February 20, 2020, 08:06:02 pmThat might be true - regarding you (and only you), but I'd expect the following scenario:VAT and duties remain unpaid, shipping costs back to Israel are unpaid, so the package is turned into a COD package (COD amount=VAT+duties+COD shipping costs), which will go through customs in Israel, requiring payment of VAT and maybe duties *again*, this time based on the imvoice amount plus the COD amount.Better be prepared for some very angry emails from the seller.(Maybe you are lucky and they managed to get this sorted, but he above happened, as being told by a reliable source.)

author: lukego
author_rank: Frequent Contributor
timestamp: February 23, 2020, 03:29:10 pm
content: Quote from: blueskull on February 23, 2020, 02:19:53 pmHere is what I don't get. The chip was designed with lots of IOs as a selling point. Are they not supposed to actually be used? Are other people not actually using them? Are there not a bunch of suitable boards already designed, living as CAD files on people's hard disks around the world, that one could potentially get access to for the purposes of fabrication?I understand the skepticism that a newbie could design a board to successfully put all the IO to use. Fine. But are we talking about it not being possible at all, even for experts with resources? Why did Xilinx bother to put so much IO onto the chip in that case if it can't be used?

author: OwO
author_rank: Super Contributor
timestamp: February 23, 2020, 04:54:50 pm
content: Ultrascale+ boards are usually at least 16 layers. If you actually break out every single one of the IOs, blind vias are practically required to not interrupt the ground and power planes. This is what the ground plane looks like with vias on it:The slit of ground plane between vias is only 0.3mm wide, and the via pitch is 1.0mm. That means you have around 30% of the current carrying capacity as without the vias. You can parallel several power and ground planes if you have lots of layers.If you can't afford 16+ layer boards with blind vias the only way to still get a usable board is to not break out most IOs. On 4 layers I can probably break out ~50 IOs + half of the MGT lanes and still have an acceptable PDN. Using all the MGT lanes is not possible until you get to 8 layers.

author: lukego
author_rank: Frequent Contributor
timestamp: February 23, 2020, 07:39:33 pm
content: w00t! I have the cards in my hands nowThanks for the words of caution about customs risks, I do think that I'd been underestimating the downside risk of returning the package and so I swallowed my pride and paid the cost of having left my VAT-ID off the import invoiceHere's a first quick tour of one:I reckon the next steps will be:Chuck it in a PCIe slot and see if it explodes.Try to prod it from a Linux host using the Mellanox tools for this card (supposed to be able to program the FPGA.)Try to prod it over JTAG.First I'll need to setup a PC for testing scavenge PCIe cards in. I don't want to take chances with the machines I have as they're all dual-socket Xeon from various projects in the past. I've picked up a $150 Intel NUC with an external PCIe slot that connects into the chassis via M.2 and a couple of riser adapters. That should make it simple and safe to try new PCIe cards. Just need to get that running somehow e.g. booting from a USB stick or better a PXE image.I'll take a poke at this next weekend...

author: lukego
author_rank: Frequent Contributor
timestamp: February 23, 2020, 07:44:43 pm
content: Quote from: blueskull on February 23, 2020, 03:55:19 pmOK. That sounds okay to me. Perhaps I an approach this incrementally. First just power and JTAG, then a 25G ethernet port, then a 100G ethernet port, etc, until we have 8x100G ethernet in a couple of years' time.Just browsing the web it seems like you could get 10pcs 16L PCB prototype for ~$1000 ballpark which is real money but not unreasonable if you had an advanced design that you were confident of & were creating dev boards that would cost ~$5k per piece off the shelf i.e. anything with this kind of FPGA.

author: lukego
author_rank: Frequent Contributor
timestamp: February 23, 2020, 07:48:34 pm
content: Quote from: blueskull on February 23, 2020, 03:55:19 pmIt's very interesting that you say this!The main project that I want to do, aside from this distraction with the UltraScale+, is to make a 1x10G NIC using a Lattice ECP5 and a microsemi XGMII PHY. The SerDes on the ECP5 would do 4 x PCIe Gen2 and then ordinary I/O to talk XGMII to the PHY. I have the chips and "just" have to design the board and write the gateware. This design is based mostly on advice on twitter but seems reasonable to me and I'd *love* a hackable 10G NIC with a BOM of <$100 per port.

author: OwO
author_rank: Super Contributor
timestamp: February 24, 2020, 04:08:26 am
content: In terms of transceiver routing, 6Gb/s is trivial, 10Gb/s is doable, and 15Gb/s is pushing it (on FR4). If you really need the highest possible speeds, I recommend only routing the outermost transceiver pairs and only on the top layer, which means using at most 1/4 of the total lanes in each quad:This way there are no layer transitions, and with some luck you *might* get a 20Gb/s transceiver going (assuming you know how to design an RF board properly). Up to 6GHz (10Gb/s) it's possible to design transitions by guessing and still get good results, but beyond that and unless you keep to some strict design rules (e.g. for RF I use 4 ground vias around each transition with a set distance), you have to do EM simulations. Without at least two ground vias around each transition, the space between planes forms a waveguide and may or may not fuck you up (which requires simulation to check for). If you try to break out 1/2 of the lanes in each quad there is already no space for 2 ground vias per transition.

author: brabus
author_rank: Frequent Contributor
timestamp: June 16, 2020, 02:26:47 pm
content: Any news?

author: lukego
author_rank: Frequent Contributor
timestamp: June 22, 2020, 07:34:41 am
content: Quote from: brabus on June 16, 2020, 02:26:47 pmI haven't done anything physical with the boards yet. My order of Galden from Imdes.de is held up somehow, about two months now, and I'm chasing that up a little bit.Meanwhile I'm trying to come to grips with the reality of routing boards like this. Since my background is software I'm starting by reviewing the literature around escape routing algorithms that are expressed in computer science terms (more "traveling salesman" and less "ohms".) I have a general idea of the state of the art in automatic routing now and I'm slowly connecting that with practical reality and the capabilities of fabs like JLCPCB.The next concrete step for me is likely to design my first-ever PCB with a smaller FPGA (Lattice) but try to do it in such a way that it works as a stepping stone towards the UltraScale+ FPGAs. Just now my thought is to try writing code to do the escape routing but we'll see how the wind blows.Sooner or later I'll want a break from soldering and board design and then I'll probably try to power up one of the boards and see if I can talk JTAG etc with it. I have a cheap Xilinx programmer from Aliexpress waiting to try out.EDIT: This KiCad forum post references the literature that I'm picking my way throughhttps://forum.kicad.info/t/tensorflow-based-autorouter/15844/14

author: lukego
author_rank: Frequent Contributor
timestamp: September 08, 2020, 08:17:47 am
content: I desoldered the first of these chips today using hot air from a Hakko FR-830 underneath and a Quick 861DW above:The next step will be to reball it, then I'll need to design a minimal test board to solder it onto.I'll solder it using vapor phase. I have acquired a suitable asparagus pot, induction hotplate, and flask of Galden LS 230.

author: lukego
author_rank: Frequent Contributor
timestamp: October 30, 2020, 12:55:05 pm
content: Quick update! This project is far in the background but still very much alive.I have seen this same eBay seller come back offering about 50 more units of this same FPGA attached to this same kind of Mellanox NIC. I don't want any more of the chips at the moment but it's very encouraging to see that more supply exists.I have pulled the trigger on buying an Imdes Mini vapor phase oven just now because I saw them on sale this week at eleshop.eu. I'm not sure if this will obsolete my (as yet unused) asparagus pot and induction plate or whether that will still be the tool of choice for desoldering.I did see that Imdes make a vapor phase desoldering jig (https://eleshop.eu/imdes-vapor-phase-imd-dss1.html) and the method of action is to pull gently on a piece of double-sided tape attached to the chip. That method does seem pretty easy to improvise without the special jig and potentially much gentler than desoldering with hot air (as I already did with apparent but not verified success.)

author: brabus
author_rank: Frequent Contributor
timestamp: October 30, 2020, 12:59:36 pm
content: Cool!Keep us posted!Any luck using the cards as they are, without having to redesign the PCB yourself?

author: lukego
author_rank: Frequent Contributor
timestamp: October 30, 2020, 01:05:35 pm
content: Quote from: brabus on October 30, 2020, 12:59:36 pmI have not tried using the cards as they are yet but I have some new thoughts on that.I have verified that Vivado is available cheap/hourly on AWS. Fired a VM up and poked around to see what's there. This does absolutely seem to be the full unrestricted version for all devices although I have not actually used it to produce a bitstream myself yet. So there is no "thousands of dollars of licenses" obstacle to using the cards as-is.I had originally assumed that the cards would be faulty - how else would they sell for so cheap? - but I now believe this was naive and that the Digi-key list prices for the FPGAs are massively inflated compared with what customers like Mellanox would pay for the chips. I actually suspect that most of the economy of using salvage chips is not because it's used but because it gets indirect access to the volume discounts that were used to purchase the chips originally. Just based on talking with FPGA afficionados on the 'net. The cards that I have do lack serial numbers though so I suppose they are some kind of prototype.There is a connector on the side that's supposed to be hooked up to 25GbE capability on the FPGA even though the main ethernet ports go to the ConnectX instead.So one day I should just sit down and try using one of them! Quite possible they are already perfectly fine as dev boards for many purposes.

author: lukego
author_rank: Frequent Contributor
timestamp: November 03, 2020, 12:13:45 pm
content: I have some across a couple of relevant links about "by the book" rework of these BGAs.https://www.surfacemountprocess.com/uploads/5/4/1/9/54196839/j-std-033b01.pdfhttps://www.xilinx.com/support/documentation/application_notes/xapp427.pdfhttps://www.xilinx.com/support/documentation/application_notes/xapp426.pdfThe maximum recommended number of reflows for a chip is three, both generally from J-STD-033B.1 and specifically from the Xilinx recommendations.The chips received on boards will have already been reflowed one or more times, and desoldering will require another reflow, and again for reballing, and again for finally soldering onto the new product. So it seems any product built on repurposed FPGAs necessarily carries the "caveat emptor" that it has exceeded the recommended number of reflows. So - scavenged parts probably fine for prototyping and development boards but off limits for production.(Cynically I wonder whether the "three reflows" rule is intended to limit the secondary market for expensive parts but that is speculation.)J-STD-033B.1 seems to recommend that any large BGA is baked at 125C for 96 hours prior to reflow. I'm not sure how seriously to take this: I'd be very keen to do this by the book if it meant having a fully legit production process but when the max reflows limit will be exceeded anyway I'm not sure how much to care.

author: thinkfat
author_rank: Unknown
timestamp: November 03, 2020, 12:17:26 pm
content: Quote from: lukego on November 03, 2020, 12:13:45 pmExceeding maximum reflow number means the published MTBF figures are not valid any more. Baking the chips gets the moisture out. This helps reliability, chance of having a bad solder joint (important for large BGA) is reduced.

author: lukego
author_rank: Frequent Contributor
timestamp: December 14, 2020, 04:37:15 am
content: Just the next baby step: I unboxed my new Imdes Mini vapor phase oven now. I'm planning to use this to solder the FPGAs onto new boards. (I'll probably do that first and then solder the rest of the parts by hand but we'll see.)

author: ApplePi
author_rank: Contributor
timestamp: February 24, 2021, 08:33:15 pm
content: I don't know if people are actually interested in getting the cards working as is, but I ordered 2 of them for using in my capstone project. I think these cards present a really cool platform for test RISC-V SoCs and using FPGA implementations of RoCE to access NVMe devices remotely over the NIC.https://docs.mellanox.com/display/Innova2Flex/NVIDIA+MELLANOX+INNOVA-2+FLEX+OPEN+ADAPTER+CARDhttps://lenovopress.com/lp1169-thinksystem-mellanox-innova-2-connectx-5-fpga-25gbe-2-port-adapter25gbps is basically PCIe 3.0 4x which should be plenty for a single NVMe disk for a test case.

author: ApplePi
author_rank: Contributor
timestamp: March 03, 2021, 11:19:38 pm
content: Okay, I just got those NICs today, they shipped with no firmware on the device, but I managed to flash them to the stock firmware and get them to show up as NICs in windows, I'll have to try and flash FPGA bundle from linux but I haven't had a chance to mess with that yet.It seems they were wiped instead of finishing production steps? Seems like nvidia killed the product and are just trying to scrap them? The seller seems to have hundreds of them.

author: ciclonite
author_rank: Contributor
timestamp: March 30, 2022, 06:26:28 pm
content: Hi! Someone have any news on this board? I have 15 of this board and wonder if i can use it without desolder fpga .

author: mwrnd
author_rank: Newbie
timestamp: June 19, 2022, 10:05:01 pm
content: I got my Engineering Sample Innova-2 Flex MNV303212A-ADL working! Several caps were broken off but traces luckily survived.Took me a while to find a compatible combination of recent Linux Kernel, MLNX_OFED drivers, DPDK, and Vivado version.Here are my notes and a simple demo project:https://github.com/mwrnd/innova2_flex_xcku15p_noteshttps://github.com/mwrnd/innova2_xcku15p_ddr4_bram_gpioMy board originally showed up with PSID: IBM0000000018 so I began by reprogramming the ConnectX-5's firmware to that in theofficial firmware release. TheDDR4 Example Designmentioned in theUser Guidefailed to implement correctly in Vivado 2017.2, 2020.1, and 2021.1. I then tried 2018.3 which the demo was originally implemented in and it worked. As did Vivado 2020.2 and 2021.2. I then used trial and error to find a working combination of software and drivers.The board heats up very quickly. Some sort of thermal solution is critical. This is a server board meant to be used in a system with high air flow. I attached a hard drive heat sink to it and placed a 12V-1.3A blower fan from a server in line with the board.I was able to test the network ports using a Direct-Attach Cable (DAC) between them but the interfaces shut down within seconds if not well cooled.@ApplePi is correct that the board makes an excellent FPGA development platform if lack of single-ended IO is not a concern. The OpenCAPI connector has 8x RX+TX GTY Transceivers and an I2C pair.The XCKU15P sits on the ConnectX-5's PCIe bridge so unfortunately using an x16-to-x1 adapter and wiring to the PCIe connector for access to GTY transceivers will not work. However, all the PCIe AC-coupling caps between the ConnectX-5 and XCKU15P are exposed on the PCB so wiring to them may work for anyone with the 100GbE QSFP MNV303611A-EDL version that lacks the OpenCAPI connector.Here are the latest software revisions that I have found to be compatible with each other. I suggest anyone with this board try this combination first.Ubuntu 20.04.4Kernel 5.8.0-43-genericMLNX_OFED 5.2-2.2.4.0DPDK v20.11dma_ip_drivers commit-7859957 March 18, 2022Vivado 2021.2