Video pass-through Xilinx based device, general thoughts/queries - Page 1
Forum section: electronics / general

author: Silenos
author_rank: Regular Contributor
timestamp: April 25, 2021, 09:59:03 pm
content: I'm being "seduced" with an assignment: device which does pass-through real time processing of up to 4K 60fps video stream, among other things. I would like to ask more experienced people for advice.And I'm researching how doable it is in Artix/low Kintex zone and whether should I try it or incontinentlyengage contingency bailout protocoldue to avalanching costs and complexity.I had some experience with Altera devices sth like 9 years ago, so I do not consider myself a total noob; I just dind't follow FPGA world for +7 years at all, and I'm not really familiar with current hardware capabilities; started to catch up recently. I also did some offline CV work.I have already found out that 4k60fps is... non-trivial. So I prolly should stick to max 1080p 60fps full rbg or sth and enjoy implementing the system with actual margins as well as relearning the wolkflow.I consider two boards:- Nexys Video (top Artix 7)- Genesys 2 (some "-2" Kintex 7)Seems both boards can handle 1080/60 over "HDMI". Still prefer Genesys2. Worth it?Some details on project: the processing involves a bunch of different series and parallel pixel-by-pixel algorithms, convolutions, compositing, and a lot of other DSP spewed all over. Propably some frame buffering would be needed too for certain bursty stuff. But I really want to avoid buffering the entire stream in external memory, I expect it to be a choke, am I right? No stream audio processing, no external frame syncing.I'm developing the DSP and demos with opencv, with offline video, with mind of it being as FPGA friendly as possible. The thing would be fed by graphic card and/or pricey cameras (yes, two sinks, one source).Now I have some general FPGA/video questions:1. How should I handle syncing multiple parallel pipelines? Example:I split the rx video stream onto Short Pipeline (several stages) block and onto Long Pipeline (many stages) block. I need to sync the pipelines outputs to blend them pixel-by-pixel. And in the end, I need to sync this blended stream with untouched audio stream and pump them both into video tx. Should I pad the shorter stream with FIFO structures (either in fabric or in external memory)? But how can I deduct the padding FIFOs depth, as I can't tell from HDL how long the Long Pipeline is (nor I would not count flipflops in output)?[Late thought] That is not a good example but on point with at least audio stream... ultimately I could just move the parallel paths and compositing into serial path, leaving very few syncing points (audio).. and the latency is not super relevant... Hmmm.2. Are these AXI-Stream cores relevant? I mean, are these good/helpful/usable/worthy (I have such an impression)? Or is it sth like STM32 HAL: marketing imposed buggy bloatware busting timings and resources?3. How should I handle processing clock vs AXI Stream pixel clock? Should I cross domain them over FIFO or sth? Or just sample the stream with processing clock and play the Valid signal? I see that I can't really pump processing clock much higher than pixel clock.4. How much Xiling charges for the HDMI/DP cores licenses? Is it permanent, in 1k$ range? Or is it another usual "suck 10k$ up annually or gtfo"?5. Is HDCP encryption a thing? I mean can the venerable Sony camera or graphic card refuse to cooperate upon discovery of how much I despise the HDMI overlords? No, I am not going to sink Netflix or similar crap.Bonus: Vivado is really slow at loading different viewports and other things... is that a feature? Quite a shame for a C++ software.I still would like to ask in advance around about 4K viability.1. External hardware:Both boards have HDMIs hooked to ios and can do only DVI-D, thus I cannot even spawn the Xilinx HDMI rx/tx core. So it seems I'd have to cobble another board with HDMI tx/rx interface ics and slap that through FMC connector to the actual GTs, am I right?Another annoyance: there is tiny a footnote in Xilinx HDMI rx/tx ip UG: the HDMI 2.0 ip cannot be spawned on Artix and Kintex "-1" devices. Hence Genesys 2 seems to be a winner here. But how trappy can it turn out? Like, can these IPs just in themselves eat majority of timing/placement margins, even on "-2" Kintex?Nexys has just 1 DisplayPort, kinda useless, but at least 2 boards seems to have DP lanes connected to GTs. Tho I would prefer leaning towards HDMI. Ideally I would like to have both HDMI and DP sinks/sources.2. Can Kintex handle the 4k60fps max rgb stream anyway, assuming the correctly handled HDMI 2.0 or DP 1.2 (~13-15 Gbps)? Like, how can I tell what pixel clock frequency the HDMI/DP ip stream can output in function of negociated data format? Or gauge such capabilities in any other way, except of, well, building target design?[Late thought] I wonder if the source always streams the audio, or it depends on EDID queries...  So it seems I actually can tell I don't support audio. Wondering if it actually thinns the downlink.Btw can the Genesys 2 board handle any quality 4K60 on its DPs? Could't find the confirmation.3. Is USB 3.2 Gen-2-whatever a viable solution in place of HDMI/DP here? Not sure how the camera outputs video to PC over USB. Does it tunnel the video mangled with prioprietary codec over the prioprietary USB class into the proprietary driver? Or simply uses the USB physical lines, like DP can? Yes, I'm trying to shoo away from USB ideas with "buy yourself the adaptors, leave me alone".If you don't mind, I'll maybe come with more queries... sorry if I ask for something plainly obvious, I'm already drowning in that quite vast lake of knowledge.

author: asmi
author_rank: Super Contributor
timestamp: April 25, 2021, 11:12:05 pm
content: I would certainly looks past Nexus Video and onto Genesys 2. It's got fully-populated DP input and output (Nexys only connects 2 DP lanes) connected to transceivers which can do 10 Gbps each, so you should be able to go up to DP HBR3 (8.1 Gbps per lane), which is more than enough for 4k@60. It's also got 1 GB of DDR3 running at up to 900 MHz with 32 bit data bus, so 900 * 2 * 32 = 56.25 Gbps, which should be enough for double-buffering of 4k@60 stream (writing and reading in parallel).I will tell you that I didn't try 4k@60 on that board (because I don't have 4k monitor over here, and none of my clients asked for it yet), but 1080p@60 is easy-peasy, even with complicated processing and composition.There is one catch with this board - the FPGA that on a board (K325T-2) is NOT included in free WebPack license. So the board comes with a voucher for license that is device-locked to K325 (it covers any package). Thankfully, unlike Altera's licensing model, Xilinx licenses are version-limited as opposed to time-limited, meaning if let your license lapse, you can still use the last version of IDE which was out at the time of expiry forever. Actually this board is the cheapest way to get your hands on a license for K325.Finally - all of those free AXI IPs are genuinely very useful, but you can probably write more efficient HDL yourself because those IPs have to be useable for a multitude of scenarios, while your own HDL can be very task-specific. Unfortunately, DisplayPort TX/RX IPs are not free, but you can still generate a trial license and use them for prototyping and/or hardware checkout, with intention to either buy a license, or design your own equivalent modules. It's not very complicated, but you have to keep in mind that you can only get a DP 1.2 specification for free (so HBR2 is a limit - which is still enough for 4k@60), if you want anything beyond that, you will have to part with some cash.As for USB 3.2 - Xilinx MGTs don't support it officially, but I seem to recall that there is a third-party IP which does. I think using USB for video is not the best idea because it doesn't guarantee bandwidth, and in my experience the kind of bandwidth you can actually get heavily depends on many factors outside your control - like if user connects your board directly into PC USB port, or through hub, or cable extender. If anything, I would sooner consider using 10G Ethernet for that (Genesys 2 can support it via FMC addon board - either commercial, or designed by you).

author: asmi
author_rank: Super Contributor
timestamp: April 25, 2021, 11:30:18 pm
content: And final two cents - first of all, you can take Vivado, request trial versions of DP RX and TX and try fitting it into K160 device (with SG 2 and in FFV package - that's important!), which is available in free WebPack license. This way you will see what kind of resources you will need and in general estimate feasibility before you commit to spending $1k on Genesys 2 board.Another thing - if your intend to do some OpenCV processing, you might want to try Vitis HLS because it supports some subset of OpenCV, so you can enjoy writing your code in C/C++ as opposed to HDL.

author: hamster_nz
author_rank: Super Contributor
timestamp: April 26, 2021, 04:26:55 am
content: Do you have access to the HDMI standards? The later (4k capable) ones are very hard to find unless you pay $$. The higher speed formats are very different to the older 1.X protocol.I've got a Nexys Video, and it can only do DisplayPort at 2.7Gb/s per channel. Only two channels are used on the mini DisplayPort, so that's 540MB/s tops, and RGB 444 1080p60 is 447MB/s. The Gensys2 is a better option for DP. The HDMI on the Nexys Video is connected to the standard SELECTIO SERDES, so in theory limited to < 1.25Gb/s, so that is 125GHz Pixel clocks if you use 444 RGB (or YCC).Also, note that the Gensys2 Vivado license will version-lock you to the version and host you install it on. You can upgrade Vivado for about a year until you are stranded on old releases node-locked to old hardware. I would only use it on Linux, where the license can be moved along with my PCI NIC card as the PC is replaced.

author: asmi
author_rank: Super Contributor
timestamp: April 26, 2021, 04:49:33 am
content: Quote from: hamster_nz on April 26, 2021, 04:26:55 amYou can easily change MAC address in Windows. No need to move anything.

author: hamster_nz
author_rank: Super Contributor
timestamp: April 26, 2021, 06:30:30 am
content: Quote from: asmi on April 26, 2021, 04:49:33 amAnd the disk signature, that isn't used on the Linux licensing, but is under windows?(trying not to be too direct about usage for Ethtool under Linux)

author: asmi
author_rank: Super Contributor
timestamp: April 26, 2021, 11:14:33 am
content: Quote from: hamster_nz on April 26, 2021, 06:30:30 amYou get to choose what kind of ID is being used under Windows. Just select NIC ID, and no disk signature is going to be used.

author: Silenos
author_rank: Regular Contributor
timestamp: April 26, 2021, 08:02:49 pm
content: Thank you. I'm pretty relieved it is doable.Yes, I had cautiously noted the genesys2 has voucher for a Kintex device license.. and in the end it is permanent, and for all packages, wow, such generosity! And DP 1.2 spec is open yaaay!Unfortunately have to stick to HDMI too.That's why in the end I'd have prototype the boards to run it, as I obviously lack the hardware to probe such links, to buy ip cores for its rx/tx and deal with the bs royalties. Afaik they charge for it anyway, so implementing pirated spec is not an option. Unless the price tag kills it. I don't care about exact HDMI spec, screw them already.But that will happen after the running prototype would be approved of, now I can see it prolly can run with just DP and adaptors, nice!So I was also right about USB. Good. I had it talked over by Apple fanbois, along with the Thunderbolt bs. They had sorely neglected my cries over USB 2 HS, and appearantly didn't even understand the differrence between 2.0 and 3.0+ and TB. And now I'm quite sure they hadn't understood any of the shit I had being vainly epxlaining to them. Good they forgot about wifi.Thanks for the Eth 10G idea, already intrigued and queued for researchI have already selected a nice 4K display to complete my glorious pcmr battlestation, such a fitting concidence for that project. Along with acquiring the camera. But the display has to wait as I can't get a graphic card capable of driving it :C I don't expect I would bang it in 6 months anyway, so maybe by this time the damn ccs will belly up, the semiconductor crysis will milden and there won't be another war or plague... ah it was too good for a too long time.Okay. Vitis reminded me the stuff Altium tried to push ~5 years ago - their fpga dev tools. Afaik it was a waste. Is that Vitis any better?And I dont expect it can digest my current processing soft, all sweetly enchanted with C++ templaty magic, threading queues and java-contaminated patterns- opportunity to always learn sth new in corporate OOS. And have some other processing delegated to python - again, learning opportunity (and some more approachable libs).If the Vitis is viable for the code written with mind of it, I'll keep it for sth else. And I enjoy both C++ and VerilogAgain thanks for feasibility gauge. And what's the "FFV package"? And "SG-2" lead me to some DS416 DMA paper applicable to Virtex 4 oO. Did you mean PG021 AXI DMA core?

author: asmi
author_rank: Super Contributor
timestamp: April 26, 2021, 08:42:21 pm
content: Quote from: Silenos on April 26, 2021, 08:02:49 pmXilinx provides an IP for HDMI 2.0, but it's also not free, and you will need to design a board for it yourself, as Genesys 2 use HDMI transceiver chips as opposed to MGTs inside FPGA. It's not very complicated, but something you will need to keep in mind. If you want to go to that route, pls let me know, and I can help to throw together a schematics for that.Quote from: Silenos on April 26, 2021, 08:02:49 pmAs I undersand, the Thunderbolt is essentially a PCI Express 3.0 via cable, so it could be a viable option if you can get it to work. Got to dive into specs, as I have no experience with it beyond just basic reading.As for actual USB, it's only good for video if you can use isochronous transfers, which imply possible loss of data. Which is OK if your goal is to display it somewhere (because human eye will likely won't even notice a few bad pixels for a fraction of a second), but is no good if you intend to do any processing or CV - for example, a missing data packet can erroneously trip some CV algorithms.Quote from: Silenos on April 26, 2021, 08:02:49 pmThere are some limitations for the code in order to use it in HLS flow (like no dynamic memory allocation, no file IO, etc.), but if your code is just a sequence of call to OpenCV, it's actually very good because you can debug your code in C before you ever generate any HDL, which is A LOT faster.As for the performance of Vivado/Vitis - well it's written in Java, so some slowness is the price you pay for being cross-platform. If you can, you might want to try running it under Linux, it feels faster over there (though I didn't do any objective testing to see if that's the case).Quote from: Silenos on April 26, 2021, 08:02:49 pmSG-2 is speed grade 2 (medium grade, 1 being the slowest, 3 is the fastest). I meant to say FFG676 package, not FFV. Different packages have different max IO speeds, FFG is the highest performance one (MGTs up to 10.3125 Gbps for speed grade 2, 12.5 Gbps for SG 3), this package also supports running DDR3 at up to 933 MHz in a single rank mode.

author: Silenos
author_rank: Regular Contributor
timestamp: April 27, 2021, 09:46:44 am
content: Quote from: asmi on April 26, 2021, 08:42:21 pmYeah, maybe it wasn't super clear: I was aware I would have to buy full Xilinx HDMI/DP* cores licenses beyond the prototype phase. As well as the additional royalties HDMI imposes. And I also detected that both boards can't drive HDMI with GTs.I expect that development costs would explode after demonstration prototype anyway, so 1k$ for license here and there wouldn't really matter. Unless thye charge A LOT for HDMI or annually; I don't know a range of price tag. Though that case has potential to kill the HDMI entirely, if DP+adoptors are actually viable without licensing.*you let me know that DP is open at the spec, so I'll try my own DP.There are still costs of multiple board spins, and I'm already not really happy of doing big ass FPGA, ddrams, gigabit links and other mess with Kicad. And I'm yet to develop BGA process anyway. I was spoiled with company Altium and Inventor given for grantedThough I have to admit I used Kicad last time ~10 years ago, will see if it has actually matured.Thank you for the help offer. I won't bother you with it, for now.It is still to decide to even commit, and to crawl past dev board. But I keep that in mind, and keep the thread updatedBtw I prolly miss sth here: why I have troubles with finding the interface ICs for HDMI/DP past 1.4/HBR2? I mean those for ESD, conditioning, equalizing, reapeating, driving strength etc. Like those TMDS141RHAR on both mentioned boards.And I have noticed DP is driven directly by GT pins.So are the FPGA/Xilinx GTs already so capable, that they have all those "PHY" goodies built in and have no issues driving things like common cables? And is it not practical to have all these things separated at such rates? Sry, I'm yet to read through GT UGs.I mean, I'm a bit worried to expose already pricey FPGA directly to the hotplug events.Quote from: asmi on April 26, 2021, 08:42:21 pmAll the USB fuss was about that some of usecased gear (like Sony A7S III we have around) has USB 3.2sth plugs. So they could plug it pass-through my device to their usecased shiny Imac (which has nothing but very few USB and TB ports) so they can grab the stream and clap in excitement "it just works! So much cutting edge lossless quality, wow!".The point is that the device is expected to be lossless on quality, so the interfaces have to be too. I didn't even mentioned any codecs. Ideally I would prefer tapping straight onto RAW stream, but it isproprietary.My guess was that the camera already encodes the stream with XAVC/H.264/whatever to achieve compression and then pumps it through USB. I'm yet do confirm that, so that's a dead point right now. And it adds up with pitfalls mentioned by you.And in the end I may be a total idiot; if there are clients, who like it, I should do it. They are to pay for it anyway, especially the Apple fanbois. But that's a long way to decide.Nevertheless I'll look into TB viability, I am not familiar with it.Checked the FTDI solutions but their 400 mbit/s rate was disapointing.That's nice of the Vitis, though it would derail my current flow. Will keep it in mind for other future fpga fun.So the ubiquitous java strikes again.. saw somewhere it is c++. Maybe it applies to the actually juicy stuff, not gui.Quote from: asmi on April 26, 2021, 08:42:21 pmDoh I missed it with Vivado stuff, sorry for that! Thanks for pointing the traps of packagy stuff.Now, found it in DS182 (Kintex dc/ac characteristics), table 56. Not super obvious to find by package, and they don't boast with max specs "Up to xxx gbps" in the Overview Datasheet. Actually unusual. And the 1 mm ball pitch, yay, so usable!Wait.Do these things ( XC7K325T-2FFG676 ) really fly for ~2k$ per piece? Or that is just currently inflated by the cc mining bs, as I can see total unavailability right now?

author: asmi
author_rank: Super Contributor
timestamp: April 28, 2021, 03:32:02 am
content: Quote from: Silenos on April 27, 2021, 09:46:44 amSorry, but here we're talking about a project with a budget well into five USD digits, so the price of Altium license (which can be much lower than their list price if you are good at bargaining and you call at the right time) is not going to be significant. 10G+ interfaces are no joke, you can't just "wing it" without proper board-level SI simulations (at least not without a lot of prior experience), and for that I would actually recommend Orcad Professional for it's comparatively good SI tools (compared to garbage that's in Altium at this point, though they told me that they intend to improve that in v22). It is at a point when a lot of things which can be safely ignored at lower speeds, become super-critical, like fill ratio of isolator, or weave orientation. Just a single PCB prototype run will cost you $500 to $1k+ as you will need to use high-speed prepregs because FR4 is no good at such speeds.Quote from: Silenos on April 27, 2021, 09:46:44 amThere is a ton of products for that - from TI for example.Quote from: Silenos on April 27, 2021, 09:46:44 amYou can use redrivers/retimers if you wish, but MGTs in FPGAs are designed to run transmission lines directly, and in fact some of their functionality exists specifically to deal with issues such lines cause. All that they need is proper ESD protection. Remember, most serial interfaces allow or even mandate AC-coupling, so even if you accidentally apply some DC voltage to these lines, it will not damage anything.Now, for HDMI you will have to use level shifter/redriver/retimer (like SN65DP159 for example) because HDMI standard is a bit of an odd animal in the world of serial standards - not only it's DC-coupled, but it's also single-ended terminated to Vcc as opposed to differential termination which is used by pretty much everything else.Quote from: Silenos on April 27, 2021, 09:46:44 amThey have FT60x series with USB 3.0 support, they can reach about 2.5 Gbps of real world net bandwidth.Quote from: Silenos on April 27, 2021, 09:46:44 amUnfortunately, due to FPGA complexity, there are a lot of gotchas - for example if you want to reach max DDR3 speed with that Kintex, you will have to use only certain IO banks and not others. So you have to be super-careful and diligent.Quote from: Silenos on April 27, 2021, 09:46:44 amYea, if you want to buy them in singular quantities, they are Expensive (and no, mining is not a factor, they've always been that way). Though if you talk to Xilinx sales reps, you can get it for several times cheaper, depending on a volume.Or you can try your luck and buy some chips on Aliexpress - they begin at about $50 per device, but you have to understand all risks, and even if they work, I'd only use them for prototyping and never for production. I've also seen some on LSCS for a bit over $100, but they come and go quickly, so - again, no good for production.

author: Silenos
author_rank: Regular Contributor
timestamp: April 28, 2021, 02:41:49 pm
content: Well, obviously expected the cost. And it was me named "pessimism spreader", "defeatism sower".Thanks for warning of winging PCB, never was sure, where the borderline of common process is. Already put some scheming into motion to get the tools time.I'm already plowing through DP spec intermixed with GTX/GTH UG. Yeah, DP is ac coupled, and GT has the stuff.Asked for FPGA prices as I wasn't following the prices previously. Found that Kintex-7 recently got flat 25% price increase because of "older product" lel.Will see if I would be told to stay with the 4K spec.

author: NorthGuy
author_rank: Super Contributor
timestamp: April 28, 2021, 05:52:29 pm
content: Quote from: Silenos on April 26, 2021, 08:02:49 pmThe newer Thunderbolt uses USB type C connector. It also works as USB 4. I've never used USB 4, but it said to be fully compatible with USB 3 and USB 2. The D+/D- wires for USB 2 are there. Thunderbolt also said to support DP through the same connector, but I was interested only in USB, so I don't know the details.I haven't actually seen Thunderbolt in action, and I haven't decided yet whether I want to fork $1.5k for Mac with M1 processor.Isochronous transfers should be Ok. Unlike bulk transfers the packets won't be re-send if corrupted, but I don't think the dedicated Video protocols would re-send failed packets neither (although I know very little about modern Video protocols).

author: Silenos
author_rank: Regular Contributor
timestamp: May 10, 2021, 01:42:52 pm
content: Yes, DP is also isochronous. I'm greenlighted and I skip the USB stuff anyway for now. I'm reluctant enough to again deal with all the USB stacks/drivers mess even on hard cores, with USB 2.0, not talking about USB3++ on FPGA.Tough TB/eth seems more viable, it doesn't seem to be a cameras' common output.Researched ddr3 and it already looks pretty brutal to develop. Ultimately such ddr3 was top stuff of PC world not so long time ago.

author: asmi
author_rank: Super Contributor
timestamp: May 10, 2021, 02:42:24 pm
content: Quote from: Silenos on May 10, 2021, 01:42:52 pmThankfully Xilinx provides free MIG IP for DDR3, so you only need to worry about getting your pinout and layout right, and the rest will be taken care of by MIG. Read DDR2/3 chapter of UG586, there is everything you need to know about how controller works, what kind of pinout it expects and what are requirements for the layout. It's not very complicated, but there are some gotchas you need to be aware of, like using HP IO banks in case of Kintex to achieve the maximum performance.One thing I recommend is to create a simple system with MIG and Microblaze using your pinout as a quick way to verify that your pinout will work, you can later use that very system during board bringup to verify that memory works like it should. At least that's the approach I use to test newly assembled boards, and I like including Linux-capable config of Microblaze, because booting Linux by itself is a very good memory test, and building a Linux image using Petalinux is rather trivial once you do it a couple of times to learn the ropes of it. I typically don't bother flashing it into the board, instead simply load it up via JTAG and see if it works. Or you can utilize memory traffic generator to achieve the same result, or come up with some other way to test the memory - it's really up to you. Just make sure to generate enough memory traffic to properly test for all edge cases like crosstalk, and if you resort to using Microblaze, be sure to disable data cache, otherwise you'd be testing writes to cache as opposed to memory transactionsOnce you have confirmed that memory works, you can get on with your own custom stuff knowing that at least that part of your system won't cause you any headaches.

author: Bassman59
author_rank: Super Contributor
timestamp: May 10, 2021, 05:01:22 pm
content: Quote from: NorthGuy on April 28, 2021, 05:52:29 pm$1.5k? Try$699.

author: asmi
author_rank: Super Contributor
timestamp: May 22, 2021, 02:20:02 pm
content: Quote from: hamster_nz on April 26, 2021, 04:26:55 amAnd now I can practically confirm that you can indeed move the license to another PC running Windows 10 if you set the MAC address to whatever it was on the old one. Just upgraded my 8+ years old rig to a brand new system with 5950X, and old licenses work just fine. New system does P&R more than 2 times faster than my old one (i7-3930K HEDT with 4-channel memory controller).

author: Silenos
author_rank: Regular Contributor
timestamp: May 24, 2021, 08:18:58 am
content: So, Linux Vivado is preffered to have the licenses "handling" covered, just in case?I'm already grievously making peace with the looming windows 10 installation, had it coming for all the years. So yeah, throwing another linux boot a the same time would do.

author: asmi
author_rank: Super Contributor
timestamp: May 24, 2021, 06:09:34 pm
content: Quote from: Silenos on May 24, 2021, 08:18:58 amNo need for a Linux if you don't want to. I did it in Windows 10. I edited my previous post to make it more clear.