Kintex-7 FPGA board with DDR3 memory and pci-e - Page 1
Forum section: electronics / general

author: OwO
author_rank: Super Contributor
timestamp: March 04, 2019, 08:48:21 am
content: I'm thinking of designing something like this since I can get the chips for pretty cheap. 3 years ago a 160k LE FPGA with pci-e x8 for $40 and 100k LE FPGA with pci-e x4 for $20 would have been a pipe dream.I'm trying to brainstorm possible applications: pci-e compute card? software defined radio? The trouble is all these rely on being integrated into a PC which is bulky and ex🅱ensive. I would like a e.g. standalone SDR system with its own SoC running Linux, which would be able to act as a network gateway as well as something I can log in to remotely to do analysis of the radio spectrum. I'm already doing that with a Zynq, but the hard memory controller there is limited to 1GB of total ram and only 32 bit width which is really limiting.One of the things I'm working on is computing huge FFTs of data from a SDR receiver for detailed spectrum analysis (in realtime). For example a size 64M FFT would require >512MB of ram just for the input buffer. Maybe a good solution is a Kintex board with DIMMs and a softcore running Linux? The softcore isn't doing anything intensive, it just coordinates things and allows remote access. Or maybe a separate SBC with a slow (USB) interface to the FPGA board? Some SBCs actually have pci-e but those would cost about as much as the FPGA.Anyone got any more ideas?

author: dmendesf
author_rank: Frequent Contributor
timestamp: March 06, 2019, 03:58:36 am
content: I'd like to know we're you can find such low prices.A cheap pci-e dogs card is a long dream of mine. Never saw one for less than 200usd with decent chips from altera or xilinx.Enviado de meu moto g(6) plus usando o Tapatalk

author: blacksheeplogic
author_rank: Frequent Contributor
timestamp: March 06, 2019, 05:31:36 am
content: If you are willing to forgo PCIe a SOC FPGA with an ARM HPS is one option that's relatively low cost. If PCIe is as must, I've seen some kits that use a small SOC and PCIe x4 M2 adapter as the computer. None of these however are in your $40 quoted range, I just don't think your being realistic on price.

author: OwO
author_rank: Super Contributor
timestamp: March 06, 2019, 07:02:21 am
content: I've already built custom Zynq boards and they are fine for lower end compute, but the 1GB ram limit (imposed by the Zynq HPS memory controller) and only 32 bit DDR3 total width is kind of limiting, which is why I'm looking at other possibilities as well. Unfortunately the Altera SoCs are not nearly as easy to source, so I'm mostly limited to Xilinx stuff.The Artix-7 is limited to a DDR3 PHY rate of 800MT/s, so I'm more gravitating towards the Kintex-7 which can do 1600MT/s (because of its "high performance" 1.8V/1.5V-only IO banks). The smallest Kintex-7 that is easy to get right now is the 160K LE XC7K160T-2FFG676I for just under $40, and I'm thinking about whether I should get a few for future experimentations. The Artix 7 with 100K LEs go for about $20. The effort to lay out >32 bit DDR3 isn't trivial, so I should pick only one family/package combination and stick with it.I don't think it's worth it economically to build one or two because from my experience it takes at least two iterations and 3x wasted chips to get a design with lots of BGA working. The reasons being you can't test bodge fixes, and the soldering is always suspect so you end up assembling at least 2 of each design to troubleshoot any issues. But if there is a card/module design that will be reusable over and over, then it might make sense. I think the breakeven point with this project will be 5 units.

author: OwO
author_rank: Super Contributor
timestamp: March 06, 2019, 07:09:28 am
content: The basic idea I have right now is a pci-e card (not mini-pcie) with laptop size DIMM slots on it. There will need to be some kind of expansion port with lots of properly matched differential GPIO pairs so that I can add a e.g. AD9363 daughterboard for SDR experiments. I think I'll probably abuse the mini-pcie connector or some other board edge connector for the expansion port.The board can be operative with or without pci-e being connected, so I can use it both as a compute card or standalone. I've already successfully routed the Zynq DDR3 on a 4-layer board, so this should be very doable on 4 layers too especially considering the "soft" memory controller has far more flexibility in pin assignments. I estimate prototyping cost to be 2 board revisions + 3 wasted chips, or 2*$60 + 3*$40 = $240 to get a working design, so if this is a design that can be reused more than ~3 times it's worth it.

author: ali_asadzadeh
author_rank: Super Contributor
timestamp: March 06, 2019, 07:56:09 am
content: QuoteThat's great, I was considering Artix 7 for their prices in china! you enlighten me upwhere do you get your chips in china? also 676pin BGA, is it doable on 4 layer? 484 can be done for sure...

author: OwO
author_rank: Super Contributor
timestamp: March 06, 2019, 08:07:21 am
content: Quote from: ali_asadzadeh on March 06, 2019, 07:56:09 amFor prototyping I just get them from taobao.https://item.taobao.com/item.htm?spm=a1z0k.7386009.0.0.63775eeavoMBkB&id=545807301276&_u=t2dmg8j26111Notice the words "可直拍". This is important because that means the list price is the correct price (rather than having to message the seller for the real price).For actual production you will want to find the upstream dealer, both to get better guarantees of consistency and usually better prices.So far I've only done 400-BGA on 4 layers, but pin count isn't a big problem because you don't need to route all the I/Os, only what you need. So the real limit isn't how many pins but rather how many I/Os you want to break out and how wide the DDR3 is. In fact given the same number of broken out I/Os higher pin count is easier because you have more "channels" between balls to route the signals out through.

author: ali_asadzadeh
author_rank: Super Contributor
timestamp: March 06, 2019, 08:53:47 am
content: QuoteThanks for the info, how do you find the upstream dealer? Also taobao does not let us search nay more, it says you should register and it needs a china cell number

author: aldi
author_rank: Contributor
timestamp: March 06, 2019, 06:27:56 pm
content: How can the XC7K160T-2FFG676I sell for only 40 dollar, the DigiKey price is 400 !?

author: asmi
author_rank: Super Contributor
timestamp: March 06, 2019, 08:34:32 pm
content: Quote from: aldi on March 06, 2019, 06:27:56 pmThey are very likely counterfeit. if you search on Aliexpress, you can also find some lots, but Xilinx logo isn't visible on either of them. I'm itching to order a couple (I'm just a hobbyist so don't really care if it's genuine as long as it works), but kind of afraid if customs catch the counterfeit and I get into troubles.

author: OwO
author_rank: Super Contributor
timestamp: March 07, 2019, 02:48:18 am
content: Most likely either new old stock, surplus that have been dumped onto the market, or just reballed used chips. I've seen listings for used Zynqs that still comes with the PCB attached. In my case my main concern would be continued supply, because in this case the taobao prices are lower than the average market price quoted on hqyun, meaning once stock runs out likely no supplier can supply at these prices anymore. The market prices are still far lower than digikey prices, but around 1.5x what they go for on taobao currently.Counterfeit FPGAs do not exist, it is too much work to reverse engineer the bitstream format and it makes more sense to just sell your own line of FPGAs like anlogic is doing.

author: NorthGuy
author_rank: Super Contributor
timestamp: March 07, 2019, 04:42:16 am
content: Quote from: asmi on March 06, 2019, 08:34:32 pmIf you decide to try, let us know what you get.

author: asmi
author_rank: Super Contributor
timestamp: March 07, 2019, 05:50:41 am
content: Quote from: NorthGuy on March 07, 2019, 04:42:16 amI've ordered a XC7A50T-2FTG256I for ~50 CAD (which is half of DK's list price), as I actually have some spare empty boards for this footprint, so I can put it on once it arrives and test it immediately. It's ePacket, so it should be here in a couple of weeks. If the part will turn out OK, I will order a few K160's and design a board for it. PDS is going to be quite a pain as it can easily eat 10 Amps of current over Vccint, plus about 3 Amps for MGTs. Getting 8x10Gbps MGTs for a 100 CAD would be an awesome deal if it works out!

author: OwO
author_rank: Super Contributor
timestamp: March 07, 2019, 05:52:45 am
content: Quote from: NorthGuy on March 07, 2019, 04:42:16 amI've bought Zynq-7010s twice from taobao, 2 chips each time, from different sellers. About $13 each. All the chips test out fine and all the functionality seems to work, but I do suspect they may be reballed chips.

author: ali_asadzadeh
author_rank: Super Contributor
timestamp: March 07, 2019, 06:11:44 am
content: QuoteThanks for sharing, which website or app is this picture? I can use QQ

author: nick_d
author_rank: Regular Contributor
timestamp: March 07, 2019, 10:33:32 am
content: My application for this is to implement MiniSAT in Verilog and use for massively parallel SAT solving. See "glucose syrup" solver to understand about parallel SAT. This idea needs embedded SRAM in the FPGA and would benefit from a Linux SoC to supervise. Not sure if it benefits from PCI-e but possibly from SDRAM.cheers, Nick

author: anovickis
author_rank: Contributor
timestamp: March 12, 2019, 11:40:54 pm
content: usually these are reballed pulls, not counterfeit

author: technix
author_rank: Super Contributor
timestamp: March 13, 2019, 04:26:26 am
content: Another idea is to use a separate PCIe interface chip like WCH CH368. That chip provides a single 32-bit PCIe 1.0 x1 device interface over a 32-bit parallel bus, allowing you to build whatever you want using cheap FPGA chips like Spartan-6 or Zynq 7010.

author: mrflibble
author_rank: Super Contributor
timestamp: March 13, 2019, 04:35:58 am
content: Quote from: nick_d on March 07, 2019, 10:33:32 amInteresting! Any particular design you're planning to use for the fpga implementation?

author: OwO
author_rank: Super Contributor
timestamp: March 13, 2019, 04:37:41 am
content: Quote from: technix on March 13, 2019, 04:26:26 amThe main issue with that is the bandwidth; if you do the math even a x8 pci-e interface (gen 2) has only 4GB/s of bandwidth each way, which is about the same as the memory bandwidth on a Zynq 7010 (DDR3 1066 32-bit). I would consider only x4 and above to be worth it, and x16 is ideal. Unfortunately even the Kintex-7 only supports x8, which means DDR3/4 memory on the card is mandatory if you want to do any real processing. I think up to one channel (64 bits) is routable on a 4 layer board, and two channels (128 bit total) might be doable on 6 layers. On the Zynq-7010 the bottleneck for FFT performance is the DDR3 bandwidth even with the most memory bandwidth efficient algorithm known (Bailey's 2-pass FFT algorithm).

author: legacy
author_rank: Super Contributor
timestamp: March 13, 2019, 10:20:12 am
content: isn't there anyminiPCIprototyping card? I know there is a miniPCIe kit

author: asmi
author_rank: Super Contributor
timestamp: March 13, 2019, 12:20:42 pm
content: Quote from: OwO on March 13, 2019, 04:37:41 amEach PCIe Gen 2 lane provides 5 Gbps of bandwidth, so x8 can theoretically provide 5 GBytes/s. But Kintex MGTs have a max line rate of 10 Gbps (depending on package and speed grade), so they can be used to implement PCIe Gen 3 connection, but in this case you will have to do it all by yourself as built-in PCIe block only supports Gen 2. There are commercial PCIe Gen 3 x8 IP cores for Kintex, so it's totally possible from technical standpoint. This will give you max bandwidth of about 8 GBytes/s.

author: legacy
author_rank: Super Contributor
timestamp: March 13, 2019, 12:29:12 pm
content: Quote from: mrflibble on March 13, 2019, 04:35:58 amnever heard aboutSATbefore, new thing learned (thanks), but now the question in my head is: for what?

author: mrflibble
author_rank: Super Contributor
timestamp: March 13, 2019, 01:05:35 pm
content: Quote from: legacy on March 13, 2019, 12:29:12 pmSolving minesweeperof course.

author: technix
author_rank: Super Contributor
timestamp: March 13, 2019, 03:10:51 pm
content: I have a feeling that for your specific uses going with high end graphics cards can be cheaper than FPGA, and it is less export controlled. If Taobao pricing is to be trusted, you can buy two or three nVidia RTX 2080 Ti cards or half a dozen AMD Radeon Vega 64 cards for the price of a single high gate count Kinetix-7 development board with PCIe x8 interface. Also for nVidia they provide pre-tuned, manually optimized implementations of some common algorithms like FFT as concrete CUDA libraries for those high end cards.The FPGA board used in a system that has a high end GPU only handles interfacing the PC to high speed ADC and DAC, allowing you to use a lower gate count, slower and cheaper part with either less internal SerDes or an external PCIe interface chip.

author: NorthGuy
author_rank: Super Contributor
timestamp: March 13, 2019, 04:01:26 pm
content: Quote from: OwO on March 13, 2019, 04:37:41 amOne "channel" takes over 100 pins. If you try hard, you may be able to attach 2 channels with 5 banks, most likely 6 - it's 300 IO pins. Any you'll have to route most of the pins in these 6 banks. If you look at pinouts, they're about 10-row deep. I don't think 6-layers is easy, if possible at all.Also, there's a bank mismatch - you have 3 HP banks and 5 HR banks.

author: asmi
author_rank: Super Contributor
timestamp: March 13, 2019, 05:01:11 pm
content: Quote from: NorthGuy on March 13, 2019, 04:01:26 pmYea, I have my doubts about this as well.

author: OwO
author_rank: Super Contributor
timestamp: March 13, 2019, 06:14:00 pm
content: I've already routed the Zynq 32-bit DDR3 on a 4 layer board, and that one was extra hard because of the fixed pin assignments and you can't swap bits in the address bus. There was also the issue of routing each address bit to two chips. I used two signal layers, a ground plane, and a VCCDDR plane. The design passes all memory tests at the maximum supported speed which is 1066 MT/s. With DIMMs and soft controllers there is none of that to deal with, you only need to route each bit to a supported IO and each address bit only need to be routed to one pin. I'm not going to be doing multi-rank memory. If you keep the routing short and use 2T timing you can get away with no termination.The DDR pins on the Zynq are also up to 5 rows deep and I managed to get it all routed. The 1.0mm pitch of the Kintex also means more channels between vias (every 3 vias can have a large channel passing two signals and a small channel passing one signal).I had a quick look at the pinout of the Kintex and indeed you can not go to 2 channels because there aren't enough HP bank pins. 1 channel looks very doable on 4 layers.Edit:Meanwhile I'm kekking hard:https://forums.xilinx.com/t5/Embedded-Processor-System-Design/Zynq-PCB-routing/td-p/369839QuoteQuoteQuoteQuote

author: asmi
author_rank: Super Contributor
timestamp: March 13, 2019, 07:10:44 pm
content: Quote from: OwO on March 13, 2019, 06:14:00 pmDo you mind publishing gerbers for that routing? I would like to see that.

author: OwO
author_rank: Super Contributor
timestamp: March 14, 2019, 03:35:33 am
content: Here are the design files for the Zynq SoM. Rename .pcb.txt to .pcb and .sch.txt to .sch before opening with gEDA.Top layer overlaid on in1:in2 overlaid on bottom layer:You can see the bottom is split into two regions, a VCCDDR plane region and a GND plane region. The DDR signals on in2 travel referenced to the VCCDDR plane always.The bottom view showing all the stitching caps:The stackup is JLC 0.1mm prepreg, 1.2mm total thickness.EDIT: added gschem symbol library. Copy the symbols to your geda symbols directory before opening schematic.

author: scatha
author_rank: Regular Contributor
timestamp: March 14, 2019, 08:52:35 am
content: Quote from: technix on March 13, 2019, 03:10:51 pmThis is a good point, with modern GPUs a 64M point FFT is considered to be pretty small. Unless you need the low latency it is invariably better to do this sort of signal processing on the GPU if you can, it is just so much easier (and often faster) than on an FPGA. Getting the samples *to* the GPU is the time-consuming part, so if you develop a cheap sampler capable of DMA-ing to the GPU that would be useful to a lot of folks.

author: OwO
author_rank: Super Contributor
timestamp: March 14, 2019, 09:07:26 am
content: FPGA dev board prices don't really mean much, the pci-e ones are all way overpriced because of low volumes. Last I checked a decent GPU (used!) will cost as much as the development cost of a Kintex-7 board (several prototypes + chips). Don't forget power consumption as well, as the applications I have in mind will be doing DSP 24/7, and the >100W power draw of a GPU will give you a surprise on your electricity bill real fast.

author: technix
author_rank: Super Contributor
timestamp: March 14, 2019, 05:16:58 pm
content: Quote from: OwO on March 14, 2019, 09:07:26 amWhen did you check it last time? GPU prices took a nose dive recently as part of the cryptocurrency crash. Also the point I was trying to make is that unless it is latency sensitive you can get away with a decent GPU and a very cheap FPGA (something on the caliber of XC6SLX9-2TQG144C) board that uses external PCIe interface chip, allowing you to pull off designs that does not even use any external memory. This means instead of a 6-layer board with DDR3, you can pull it off with a 2-layer board and does not even use any BGA packages.

author: asmi
author_rank: Super Contributor
timestamp: March 26, 2019, 01:24:46 am
content: Quote from: NorthGuy on March 07, 2019, 04:42:16 amI've received the chip today. Don't have time to solder it yet, however I checked device's serial number through Xilinx "Device Lookup" service - and it turned out to be speed grade 1I instead of 2I as promised. Date code is 1841 (which I assume means 41st week of 2018). Other than wrong speed grade, it seems OK (see attached photos).

author: ali_asadzadeh
author_rank: Super Contributor
timestamp: March 26, 2019, 06:00:40 am
content: QuoteWhere did you get them?

author: Scrts
author_rank: Frequent Contributor
timestamp: March 26, 2019, 05:47:56 pm
content: Quote from: ali_asadzadeh on March 26, 2019, 06:00:40 amOrdered from Iran, I guess.It's written in the previous posts...

author: NorthGuy
author_rank: Super Contributor
timestamp: March 26, 2019, 07:15:49 pm
content: Quote from: asmi on March 26, 2019, 01:24:46 amQuit fast. It may be hard to distinguish -1 from -2 grade, may be only by special tests. Of course, it only matters if it works after you solder it

author: asmi
author_rank: Super Contributor
timestamp: March 26, 2019, 08:03:05 pm
content: Quote from: NorthGuy on March 26, 2019, 07:15:49 pmI always order at Aliexpress with delivery via ePacket as it's trackable and quite fast (normally 2-3 weeks to Ontario, might be few days faster for more westward locations).For speed grade - since 2017 Xilinx stopped laser-etching speed grade onto the package, right now the only way to find it out without soldering chip onto the board is to use their "Device Lookup" service by reading 2D barcode that is etched on the package (bottom-right corner of package). So it's kind of complicated now, especially since you need Xilinx account and their approval to use this service.Performance-wise there is a pretty significant difference between -1 and -2, for example MGT at -1 can only reach 3Gbps vs 6Gpbs for -2, DDR3L max speed of 667 vs 800, etc.I've asked the seller about the wrong speed grade, so will see if it is an honest mistake, or intentional misinformation in the listing.

author: OwO
author_rank: Super Contributor
timestamp: March 27, 2019, 05:13:51 am
content: I scanned my Zynqs but none show speed grade info on either the Xilinx Go app or the web app. It was a speed grade 1 though so I'm not that concerned.

author: OwO
author_rank: Super Contributor
timestamp: March 27, 2019, 05:30:30 am
content: OK so this is where it gets sketchy. I tried to look up the device DNA (read using JTAG) withhttps://xapps9.xilinx.com/2dbarcode/but it says "no data found" for all 3 devices I tested.Seller 1: (date codes 2018)38120A1E0325343F2812087254282E1FSeller 2: (date codes 2014)2A0D8D1261843C4B(I have one more from seller 2 that I haven't soldered yet)The register dump of the last device:Quote

author: Axk
author_rank: Regular Contributor
timestamp: March 27, 2019, 10:12:03 pm
content: Sorry for an off-topic question guys, but how do you control DDR3 with the FPGAs?Do they have built-in hard core controllers?To use the controllers do you have to pay?Are there trial versions for prototyping?

author: NorthGuy
author_rank: Super Contributor
timestamp: March 27, 2019, 10:28:36 pm
content: Quote from: Axk on March 27, 2019, 10:12:03 pmSome of the FPGAs have built-in DDR controllers. Some have "helper" blocks. Xilinx has free DDR3 component called MIG (Memory Interface Generator). Or you can write your own.