HDMI image grab with microprocessors. Is it feasible? About HDMI Demystification - Page 1
Forum section: electronics / projects

author: donkey
author_rank: Regular Contributor
timestamp: July 30, 2018, 11:09:00 am
content: Hi dear friends,This will be my blog message about my ongoing learning curve about HDMI and electronics. I will send all of the useful materials i will come up with on this topic; of which i think that will help me and to other beginners in their research.I would like to ask you, if it will be possible to grab a single frame with the use of an Evaluation Board in the link below?http://www.analog.com/media/en/technical-documentation/user-guides/UG-338.pdfby using any type of a microprocessor's pins?Any simple documentation on HDMI single frame receiving concept is also greatly appreciatedWhat i would like to achieve is to grab a single frame from my DVB-C STB's HDMI 1080HD output, with the use of any available pins of any type of a microprocessor.Thank you very much.

author: donkey
author_rank: Regular Contributor
timestamp: July 30, 2018, 01:08:01 pm
content: An ongoing learning process ...Which useful links i have found so far.To understand the HDMI cable better, let’s first take a look at all the signals travelinside one HDMI cable:HDMI 1.4 Demystifiedhttp://www.luxielectronics.com/attachments/File/HDMI_1_4_Demystified.pdfHDMI Demystifiedhttps://www.fpga4fun.com/files/HDMI_Demystified_rev_1_02.pdfHDMI 2.0 and the push for 4khttp://luxielectronics.com/attachments/File/HDMI_2_0_and_the_push_for_4k.pdfPushing the 4K/UHD Digital Envelope 18Gbps and Beyondhttps://nordost.com/downloads/Pushing%20the%204K.pdfVHDL Implementation of TMDS encoder for the transmission of video signals in serial communication (International Journal of Advanced Research in Computer Engineering & Technology (IJARCET))http://ijarcet.org/wp-content/uploads/IJARCET-VOL-4-ISSUE-4-1576-1579.pdf

author: dmills
author_rank: Super Contributor
timestamp: July 30, 2018, 02:05:58 pm
content: I am not seeing HDMi over GPIO being possible (at least not directly), the data rates are stupidly high and I would be reaching for an FPGA to do the deserialisation and frame storage, then use whatever you like to get the data over into the micro.Regards, Dan.

author: DaJMasta
author_rank: Super Contributor
timestamp: July 30, 2018, 03:32:34 pm
content: I sort of doubt this is possible as well. You could get a chip that takes the HDMI bus of CML differential pairs and converts it to a wide LVTTL bus or something that would be signal compatible with a micro, but with 1080p60 video, you're still talking 3.2Gb/s data rates.... so even deserialized running at the minimum 165MHz character rate, it would take a pretty powerful micro just to be able to read the output. This evaluation board uses a later hdmi standard with a faster data rate and includes two HDMI streams onto the single interconnect bus, increasing the speed and storage size demands. Then you have to read enough of it in quick enough succession to store a frame of it it in memory - and then have to do any processing you need to get it into a usable form (though this part wouldn't necessarily need to be fast).You could think of what you need similar to the frontend of a digital oscilloscope: you need something capable of a very fast data capture directly into memory on a trigger, then you need your micro to be able to sort through what was stored to find what you want. Because of limitations of the speed of the micro, it's probably easiest to capture everything in a very quick storage subsystem, then have your micro sort through it after the fact, rather than trying to pass all the information through the micro at the full data rate. Of course this means you will only be able to capture a second frame after the first in the time it takes for the micro to do what it needs and for the memory system and trigger to be reset, but short of the massive amount of computing realtime processing would take, it's what can be done. A Pi probably wouldn't have trouble managing it, but I don't think the GPIO pins are suited for fast data transfer, so that's probably where your bottleneck would be.

author: NiHaoMike
author_rank: Super Contributor
timestamp: July 30, 2018, 03:40:42 pm
content: You could get a HDMI capture device that connects over USB or Ethernet.

author: james_s
author_rank: Super Contributor
timestamp: July 30, 2018, 04:19:10 pm
content: There's no way this will work with a microcontroller alone. Using a suitably capable FPGA it could be done but GPIO on a microcontroller will not be anywhere near fast enough.

author: donkey
author_rank: Regular Contributor
timestamp: July 30, 2018, 06:23:00 pm
content: Thank you very much for all of your answers.I know that there are some HDMI capture cards to buy and to capture a frame.The aim of this project is to understand the correct connection and communication procedure between an HDMI receiver IC and a system (FPGA, Microprocessor etc.) to manage to grab a single frame from it.I will keep updating the post until i will find a successful way to manage to grab a single frame capture over HDMI.Thank you!

author: donkey
author_rank: Regular Contributor
timestamp: July 31, 2018, 09:30:33 am
content: What i have found so far on the subject ...TMDS33DVI and HDMI specify that the same differential signalling is used on all the high-speed lines. Differential signalling is a very old and good technique for increasing signal to noise and resitance to common-mode noise. In the olden days transmission lines using this method were called "balanced".Instead of sending the signal on one wire (referenced say to 0v), you send the signal on two wires, known as + and -. The - signal is the same signal as +, but inverted (so if + is 1, - is 0 and vice versa). When the signals are combined, "common-mode" noise, noise that appears on both signals the same, tends to be cancelled out. And when differential signals are routed for long lengths, the wires are twisted togther so their emissions also tend to cancel.HDMI needs more bandwidth than can be sent on one differential pair. So they use 3 data pairs and one clock pair, plus some auxiliary signals. (Two of the auxiliary signals are also in a twisted pair physically, but they are not TMDS33 or related to the HDMI data stream.) In total there are 19 conductors in the familar HDMI cable.HDMI Cable wiringIn DVI and HDMI the 3 data lanes and the clock lane are differential pairs using TMDS33 levels. TMDS requires 51R pullups to 3.3V at the receiver. However the differential voltages themselves are much smaller, on the order of 100 - 200mV. Smaller levels are quicker to reach and allow faster transmission rates.The presence of these termination pullups is detected by the transmitter and it may suppress data transmission until they are seen, known as "receiver detection".My 'scope is too weak to record the data faithfully at 742.5MHz clock used in 720p, but below gives you an idea of what the differential signals look like when pulled up to 3.3V and transmitting.TMDS33 signal levelsIt also shows how inadequate a 200MHz bandwidth scope is for looking at this, there are actually ten bit-times shown between the two vertical lines (roughly one 74.25MHz period). HDMI receivers (and the FPGA used here) have higher bandwidth inputs that can resolve the individual bits properly.TMDS33 ChannelsThe DVI or HDMI cable itself carries the clock pair and three data pairs that transmit in parallel.HDMI data transmissionAlthough the actual clock rate is very high, to ease carrying the clock on real cables and reduce RF emissions, the clock that is sent on the HDMI clock differential pair is 1/10th of the rate of the data on the data pairs. This makes the HDMI clock period represent one pixel period, in other words considering there are three data channels, there are 30 bits transmitted per HDMI clock period (== per pixel).A PLL in the receiver reconstructs the x10 clock and uses it to capture the data on the channels.TMDS codingsUnlike a VGA or earlier cable, there are no wires reserved to carry the sync signalling. Instead, syncs are carried as part of the 10-bits per pixel data from each channel, and they are carried differently according to what else is being sent.The reason for this is that HDMI is derived from the earlier DVI standard, which had a very simple plan for carrying the syncs using reserved symbols during the whole of the blanking period. But HDMI builds on DVI by allowing new "data islands", using a different coding scheme, to randomly be carried during blanking as well as "control periods", and makes explicit "video periods" that contain the active video data: since DVI doesn't have these extra concepts it means HDMI ended up with two (syncs cannot change during active video data) completely different ways to express HSYNC and VSYNC state in the stream.DVI + HDMI codingsThe direct logical codings in HDMI then areActive video data using 10b8b coding (providing 8b each for RGB in RGB mode)Control periods using one of four special 10-bit control codings to express the 4 possible HSYNC + VSYNC states (10b2b)Data islands using TERC4 (10b4b) codingDuring the active video region where pixel data is being sent and 10b8b coding is used, there is actually a choice of two codings per byte. Once choice has more zeros and the other more ones. The transmitter selects which coding to use per pixel based on whether it has sent more ones or zeros lately, it keeps a running count and selects the coding to keep the ratio of ones to zeros at 50:50 overall.TERC4 is a sparse 10b4b coding that is used to carry both HSYNC and VSYNC data and generic data such as HDMI audio samples. TERC4 has its own structure and subchannels, and inside the overall 36-byte packets sent using it, various packet types can be found (including PCM or other audio samples).Although the Conrtrol Period reserved symbols are unique, actually you can't interpret HDMI data overall without parsing what has been going on before to eg, understand you are in an active video period or data island and track it using a state machine.Types of stream sync in HDMIAt the high data rates of HDMI, skew between clock and data, or between data channels, or between differential pair members on a single channel or clock can destroy data integrity. These skews depend on the transmitter and cable as much as the receiver, and they vary somewhat with temperature.So receivers typically have to hunt for the best clock phase with the least error rate... FPGAs have some support for either delaying to clock or individual data channels by small amounts to compensate. All high speed serial buses have some need for this including DDR, SD and PCIe who call it "tuning" or "training".If we are able to collect bits with low error rate, we need to decide which on the 10 bits per HDMI clock is the first bit. This can be done by trying all 10 and finding out which exhibited the most valid control period symbols.At that point we are aligned enough we can receive the 30 bit pixel data as it was sent.The reserved control symbols are then used to align to higher level decoding state to track where we are in the raster, since they don't appear in either TERC4 or 10b8b data.Refresher on generic video timingThe structure of video streams is still basically the same as used in the first black and white TVs, with CRT type displays.Generic Video TimingBasically, there are two kinds of period, active video (black) and blanking (grey). Normally only the active video part is shown on your display, but for our purposes, we are interested in capturing ALL of it.Originally the blanking was used as the time required to swing the electron beam illuminating the CRT phosphor back to the start of the next line (on HSYNC) and back to the top left (on VSYNC) and it had no other purpose. On analogue TVs audio is sent separately on a different but related frequency. When Colour was added in analogue video, a chroma burst used to sync the phase of a chroma subcarrier was added in the back porch of each line. The sync pulses were encoded as different analogue voltage excursions not used by the video part.Digital video formats kept the basic HSYNC and VSYNC timing and the concept of blanking intervals. So they still have front and back porches and HSYNC and VSYNC durations. One of the reasons the blanking interval won't die is because it allows you to tune the refresh rate of the video at a given pixel rate without changing the active region; this is how it is possible for HDMI to do both 720p50 and 720p60 using the same 74.250MHz pixel clock... the active video area is the same but the amount of blanking pixels is traded off with the time needed to send ten extra frames per second: 720p50 has an artificially large blanking period as you can see from a real capture below.720p50 Active Region vs BlankingIt actually uses 1980px for each line even though only 1280px contain active video.Once it was decided that there was an unused blanking interval basically as padding, in a digital protocol naturally minds turn to using the data passed there for something good. On DVI it simply passes 10b2b control period codes that encode the HSYNC and VSYNC state. On HDMI you can still do that, but you have the option to place "Data Islands" in the blanking (these are the red areas in the picture above). After error correction codes are removed, these consist of a 3-byte header and 4 x 7 byte subpackets.Generic Video TimingThe three-byte header is defined in HDMI to contain a packet type, version and length information, and there are a list of packet types such as those carrying PCM Audio samples: at 24bps, the 4 x 7 bytes is enough to carry the 8 channels supporred by HDMI with 4 bytes left over.Because HDMI sends these data islands using TERC4 (10b4b) on 2 of the three channels, and including error correction, 36 bytes must be passed, each data island packet "costs" 36 pixel-times of blanking plus some overhead to start and stop a data island in the protocol. During this time, the other data channel is used to send HSYNC and VSYNC state also in TERC4.HDMI defined Data Island Packet TypesHDMI defines the following packet types... it's legal to just ignore them and act like it's DVI, if your HDMI sink doesn't need features like audio. However we are interested in audio and the other information.0x83 (SPD) is quite interesting, the source device can send a vendor and product description in ASCII. On my laptop, it sends "Intel" and "IntegratedGfx".Reference :https://warmcat.com/

author: donkey
author_rank: Regular Contributor
timestamp: July 31, 2018, 11:03:49 am
content: Uncompressed HDMI Video Input to Raspberry Pi in OpenFrameworks

author: donkey
author_rank: Regular Contributor
timestamp: July 31, 2018, 11:39:19 am
content: Dissecting HDMIDeveloping open, FPGA-based capture hardware for conference & user group recording

author: donkey
author_rank: Regular Contributor
timestamp: August 01, 2018, 08:07:43 am
content: In this video also some HDMI concepts are become more clearhttps://ksr-video.imgix.net/projects/850748/video-353869-h264_high.mp4In the video stated that a720pvideo format timings are:Active video:1280x720Total video:1650x750Hsync :40 ( Front porch: 110, Back porch : 220 )Vsync :5 ( Front porch: 5, Back porch : 20 )Interlaced :0

author: amyk
author_rank: Super Contributor
timestamp: August 01, 2018, 11:51:18 am
content: If you manage to bitbang HDMI from an MCU, you'll probably have achieved a world-first.Using HDMI-DVI converter IC, it might be barely possible since you can work at pixel clock instead of bit clock.

author: donkey
author_rank: Regular Contributor
timestamp: August 01, 2018, 01:16:29 pm
content: Thank you amyk for your reply. From the replies i already assume that the answer of my question is <<No, it is not feasible! >>I am aware of that this topic is a little bit gone beyond my question's scope but i decided to keep posting some useful matarials on this topic which i will come up with in my learning curve.I am willing to write a conclusion and an explanation of my findings to answer my own question at the end.As a beginner myself, i hope some of the materials i will post here may also help for other beginners to answer their own questions in their own similar research curve.Again, thank you for your reply.

author: donkey
author_rank: Regular Contributor
timestamp: August 05, 2018, 10:48:29 am
content: Development & Testing of HDMI 2.0 Interfaces | TektronixUnderstanding Bandwidth, Rise Time and Signal Fidelityhttps://www.ece.ubc.ca/~robertor/Links_files/Files/TEK-Understanding-Scope-BW-tr-Fidelity.pdf

author: donkey
author_rank: Regular Contributor
timestamp: August 05, 2018, 08:38:24 pm
content: Video Connectivity Using TMDS I/O in Spartan 3 FPGAshttps://www.xilinx.com/support/documentation/application_notes/xapp460.pdfImplementing a TMDS Video Interface in the Spartan-6 FPGAhttps://www.xilinx.com/support/documentation/application_notes/xapp495_S6TMDS_Video_Interface.pdfSpartan 6 1080phttp://hamsterworks.co.nz/mediawiki/index.php/Spartan_6_1080p

author: hendorog
author_rank: Super Contributor
timestamp: August 05, 2018, 10:13:38 pm
content: I know nothing about HDMI, but I wonder if some sort of equivalent time sampling arrangement might allow you to progressively grab a frame.

author: Lukas
author_rank: Frequent Contributor
timestamp: August 06, 2018, 01:43:40 am
content: Quote from: hendorog on August 05, 2018, 10:13:38 pmI did that for VGA some years back, sampling one column per frame using the builtin ADC.After deserializing HDMI to parallel video, the same thing should be possible by the timer triggering a DMA transfer from a port to memory.Apart from that, some MCUs like the bigger STM32 can interface to cameras, using an interface comparable to parallel video. Given the pixel clock isn't too high, that may work as well.

author: hamster_nz
author_rank: Super Contributor
timestamp: August 06, 2018, 01:58:48 am
content: I've read extensively on DVi-D, HDMI, DisplayPort and so on, and know most of the older standards well (the newer ones like HDMI 2.0 are under NDAs).I'm happy to clarify any questions you might have.

author: BrianHG
author_rank: Super Contributor
timestamp: August 06, 2018, 03:47:38 am
content: You can always buy a SBC with HDMI input and HDMI output. Like one of these:http://linuxgizmos.com/snapdragon-600-sbc-offers-an-hdmi-input-and-dual-hdmi-outputs-for-99/Grab frame, grab video, even realtime compress grabbed video...

author: Zucca
author_rank: Unknown
timestamp: August 06, 2018, 11:07:43 am
content: Nice post!I am still dreaming to convert this old crapinto HDMI without an external ADC sampling with like those ones:I can't stand the idea to take digital data (source VGA) and then get the analog to convert again in digital...It would be so nice to re-design this Keysight DSOXLAN LAN/VGA moduleinto a LAN / HDMI one or to be able to tap in the VGA output circuits and mod them into HDMI output onesVGA must die!

author: james_s
author_rank: Super Contributor
timestamp: August 06, 2018, 03:13:10 pm
content: I like VGA, there are plenty of old displays around that have it and it pretty much always just works. It's a bit like RS-232, it's obsolete but ubiquitous and easy to use, so it still has value. Aside from the HDCP hassle, HDMI is generally superior and I use it if it's available but I'd still like to see VGA stick around.

author: firehopper
author_rank: Frequent Contributor
timestamp: August 06, 2018, 06:10:19 pm
content: Quote from: zucca on August 06, 2018, 11:07:43 amactually vga is analog from what I understand, the rgb and sync signals are all analog, thats why the need of a A2D converter.

author: BrianHG
author_rank: Super Contributor
timestamp: August 06, 2018, 06:19:54 pm
content: Quote from: firehopper on August 06, 2018, 06:10:19 pmYes, though the H&V syncs are digital, they are just clocked with the Analog DAC.And, cheap small sample conversion boxes are crap and produce a crap HDMI image unless you go for a full blown quality scaler, doing a 1:1 outputting of the true source resolution & real high end VGA cables. Otherwise, your text will be mush.

author: donkey
author_rank: Regular Contributor
timestamp: August 15, 2018, 06:14:49 am
content: Thank you for your courtesyhamster_nzif you are the guy who own this page (http://hamsterworks.co.nz/mediawiki/index.php/Main_Page). You are my man! Great recourses for anyone who requires. Before asking any further questions, firstly i decided to try to clarify the concepts of the underlying technology by myself. I am a total beginner and i am even struggling with the elementry concepts of the technology. I have to try to answer some of my dumb questions on my own, to ask some less dumber questions to the forum.If you are the guy who own the link(http://hamsterworks.co.nz/mediawiki/index.php/Main_Page). I want you to know that you did a such great work. Keep that great work and long live. I am also subscribed to your Youtube channel.Thank you!

author: donkey
author_rank: Regular Contributor
timestamp: August 15, 2018, 06:19:04 am
content: A great productBrianHG. Thank you for your recommendation!

author: hamster_nz
author_rank: Super Contributor
timestamp: August 15, 2018, 06:30:43 am
content: Yeah, that's me..... :-)

author: donkey
author_rank: Regular Contributor
timestamp: August 15, 2018, 07:53:46 am
content: I'm honored to meet you,hamster_nz

author: donkey
author_rank: Regular Contributor
timestamp: August 15, 2018, 07:54:30 am
content: Ultra HD HDMI to MIPI CSI-2 (Camera Serial Interface) converter chipsethttps://toshiba.semicon-storage.com/info/docget.jsp?did=28743&prodName=TC358870XBG

author: donkey
author_rank: Regular Contributor
timestamp: October 18, 2018, 08:16:52 am
content: HDMI Oscilloscope requirementsCredit goes toGuenter LehmannfromAnalog Devices, answer onQuestion :"Where do8 GHz minimum bandwidth and a sampling rate of 20 Gsample/s for HDMI 1.4b signal12.5 GHz minimum bandwidth and 40 Gsample/s for HDMI 2.0 signalrequirements are comming from ?"Answer :1.) HDMI data rate = 10 * PixelRate * (ColorDepth /8)1080p60 (pixelrate=148.5MHz), for 8 bit color depth -> data rate = 1.485 Gbit/s on each TMDS data pair12 bit color would give us -> data rate = 2.2275Gbit/s on each pair.The 10x multiplier comes from the 8b10 encoding use to push one color plane down a TMDS pairFor 1080p120, 8 bit color depth -> 10 * (2 * 148.5MHz) * (8/8) = 2.97GBits/s (3G)For 4Kp30, 8 bit color depth -> 10 * 297MHz * (8/8) = 2.97GBits/s (3G)For 4Kp60, 8 bit color depth -> 10 * 594MHz * (8/8) = 5.94GBit/s (6G)Or 4Kp60, 8 bit color depth -> 40 * 148.5MHz * (8/8) 5.94GBits/s (6G)Note for 4Kp60 the TMDS clock line runs at 148.5MHz but the multiplier is changed to 40x2.) When we look at square waves with scopes, the bandwidth of the scope needs to include the 3rd and 5th (and sometimes 7th) harmonic (Fourier) of the fundamental frequency you are trying to measure. Ideally we’d like the scope bandwidth to be 10x the fundamental frequency we are trying to measure else we’ll see a distorted image on the scope. So for 1080p60 where the fundamental is 1.485GHz, the scope bandwidth needs to be greater than 1.485GHz * 5 or 7.245Ghz. Digital sampling scopes need to have a sampling rate 2 x bandwidth (Nyquist). The same applies to HDMI 2.0, just faster.I wrote this guide as a simple set of rules PCB designers can use when laying out HDMI traces. The key to HDMI routing is to keep the trace flow smooth, no stubs or tight turns and keep the differential impedance to 100 Ohms. Eye patterns are used to understand why your routing doesn’t work or meet HDMI specifications (theoretical background). Many layout people don’t have signal integrity tools to simulate the eye pattern and test engineers don’t have test equipment to measure the eye pattern.Hope this helpsGuenter LehmannSr. Application EngineerAnalog Devices