Reading data from MIPI CSI-2 camera sensor - Page 1
Forum section: electronics / general

author: malkauns
author_rank: Contributor
timestamp: January 28, 2017, 08:18:00 am
content: I'm trying to understand how to set up clocks and read data from a MIPI camera sensor. The sensor (Omnivision 5647) uses the MIPI CSI-2 protocol with D-PHY for the physical layer. The stage I am trying to get to is to be able to observer SoT (Start of Transmission) signals after which I can start parsing the CSI-2 protocol packets. In a small MIPI writeup located athttp://archive.eetasia.com/www.eetasia.com/ART_8800715969_499489_TA_a466fca2_3.HTMthere are 2 statements that are to be taken into consideration when trying to read data:"The high speed payload data from the transmitter is transmitted on both the edges of the High speed differential clock (DDR clock)""The high speed differential clock and the data transmitted from the transmitter are 90 degrees out of phase and with the data being transmitted first."Using VHDL and Vivado, how do I create logic to successfully read data from this sensor? I have the following code written (with notes/questions) but I'm pretty sure its wrong. It was put together based on my limited understanding and reading various other source code that perform similarly:http://pastebin.com/FGvChHisI was told that in order to derive the correct delay value I would have to sample the output clock at the rising edge. If it is not 1, decrement the delay value. If it is 1, increment the delay value. This way the delay should always be within +/- 1 of the ideal value.I have experimented with this code and tried to see how many SoT's I can detect but its very low (<10 per minute). This is probably due to random chance.Really need help on this one!

author: daveshah
author_rank: Unknown
timestamp: January 28, 2017, 10:38:28 am
content: Have a look at my implementationhttps://github.com/daveshah1/CSI2RxFor the inputs I used a IBUFDS, IDELAY and then ISERDESE2 in DDR mode. The delay of the IDELAY was tuned for best results. Also note that bytes have to be realigned as there's no guarantee how they'll come out of the serdes.Let me know if you have any further questions - I found that development and tuning took a while but in the end I was able to get a very stable implementation, which works up to 4k at at least 48fps.

author: malkauns
author_rank: Contributor
timestamp: January 31, 2017, 12:47:33 am
content: Hi Dave,Thanks a lot for your code. It was a great help. However, I have a strange problem. I can observe continuous clock signals frombit_clock_int_prebut nothing frombyte_clock. Actuallybyte_clockdoes increment a little after I load registers over SCCB but then it stops. I have the following IP's:csi_rx_hs_clk_phy:http://pastebin.com/23xn9aWkcsi_rx_hs_lane_phy:http://pastebin.com/aSP0ayBPregisters:http://pastebin.com/tr8kg01EHere is how it is connected in Vivado:http://imgur.com/a/Q0N96I just modified yourcsi_rx_hs_lane_phya little to addIDELAYCTRLandrefclk.clk_wiz_0provides a 200MHz clock torefclk.resetis low,enableis high.debug0(bit_clock_int_pre) andbyte_clock_outrun to IP's that count rising edges per second and I can see this hover around 95MHz fordebug0. I was expecting to see some changing bytes fromdeser_outand some rising edges per second frombyte_clock_outbut as I said earlier it appears to stop shortly after running. Not sure why this is happening.

author: daveshah
author_rank: Unknown
timestamp: January 31, 2017, 08:41:18 am
content: That is odd. What is your reset signal going into the PJY blocks doing?

author: malkauns
author_rank: Contributor
timestamp: January 31, 2017, 08:45:46 am
content: Nothing special. Here's a wider picture:http://imgur.com/a/iW5mW

author: malkauns
author_rank: Contributor
timestamp: January 31, 2017, 09:09:11 am
content: OMG you're right!! I used a constant to set both resets on those ip's to 0. Now i see stuff ticking!! My byte_clock is around 23.5MHz! I guess I made the mistake of thinking that the FCLK_RESET0_N was active high. I'm sure there'll be more questions soon but this should keep me busy for a while. Thanks a lot Dave!!!

author: daveshah
author_rank: Unknown
timestamp: January 31, 2017, 09:12:17 am
content: Yes my PHY blocks need an active high reset and the BUFR won't produce the byte clock if reset is asserted.I've actually wasted a couple of hours troubleshooting a similar issue with a DDR3 core before now - in that case I thought it was active high so tied it to 0 but it was actually active low.

author: malkauns
author_rank: Contributor
timestamp: February 02, 2017, 05:44:03 pm
content: And sure enough, I have another question. The problem now is that I am not getting the expected amount of SoT's. According to the MIPI doc, both lanes should present SoT's simultaneously. I've experimented with the delay value and played withbitslipto adjust the bit alignment between tests. However i'm only seeing a few SoT's (less than 10) per second which is probably due to random chance more than anything else. Shouldn't there be thousands of SoT's per second?

author: daveshah
author_rank: Unknown
timestamp: February 02, 2017, 05:56:28 pm
content: Is reset still now tied to a constant 0? I think the SERDESs don't like that and need reset to be de-asserted once the clock is stable. With my cameras the clock is lost between frames so I have a block to detect clock failure and hold the SERDES in reset until clock is stable.There's also a chance the camera isn't actually outputting any data at all; I've found Omnivision cameras can output a clock but no CSI packets if they don't like the timings. Have you checked for any activity on one of the data lines with a scope?Another issue could be signal levels as a device on the low end of CSI-2 specs wouldn't comply with the Xilinx LVDS specs; although clock and data should be at the same signalling levels so if you're getting a stable clock I don't think that's a problem.You mention bitslip; there is no guarantee of alignment at all I believe - my code looks for SoT with any alignment on all lines near-simultaneously (allowing up to 2 cycles of skew just in case). Nonetheless even if you only look with one SoT you should still see 1/8th of the expected number which is a lot more than <10 a second.Also at some point you particularly once you consider all alignments will probably find that you see false SoTs - this is when the lines are in the LP11 state inbetween packet so the LVDS receiver output is undefined, or if the code accidentally goes back in to "wait for SoT" mode during a long packet's payload due to some other glitch. If you're doing things properly you can connect the CSI-2 lines to singleended IOs as well as an LVDS input to detect the LP11 state but I haven't ever bothered with this. In my case as well as looking for SoT I also reject any packet with an unexpected data type or bad ECC in the header and that works out to be pretty robust against false triggers - I don't think I've ever noticed a visible glitch.

author: malkauns
author_rank: Contributor
timestamp: February 03, 2017, 07:40:55 am
content: Your reply got me thinking about the "no guarantee of alignment" issue which after I thought about it makes perfect sense. So I started on code to handle the case of looking for SoT's within a 2-byte window per byte_clock cycle. Results are now very encouraging. I am seeing around 3500 matching (both lanes) SoT's per second although after a short time they mysteriously dropped to 150/sec. The SoT frequency seems to vary by quite a bit over time. Not sure what is going on there. Is this what you would expect?

author: daveshah
author_rank: Unknown
timestamp: February 03, 2017, 09:19:57 am
content: Just in case the alignment is slightly off it might also be worth considering SoT on both lanes with up to 1 byte clock difference.The mysterious decreasing and changing frequency does sound a bit odd. Even 3500 feels a bit on the low side, it appears from those registers you're outputting 1296x730. There should be one packet and thus SoT on both lanes for each line sent (plus frame start and end); so at 30fps you should be expecting almost 16000 every second.I have found the SERDESs to be a bit finnicky when it comes to reset and timing. Have you got some kind of input delay constraint set up (I found that just having the constraint eliminated some glitches when I was playing around), and are there any timing errors? What does the SERDES reset logic look like now?

author: malkauns
author_rank: Contributor
timestamp: February 07, 2017, 07:56:11 am
content: Hi Dave,Sorry for not getting back earlier and thanks for all your help so far. I implemented your idea of searching for SoT's on both lanes within a 3 byte window but it doesn't seem to improve things. The SoT frequency goes all over the place. For reference here are my sources:mipi_phy_stats:http://pastebin.com/XZFV8bk7csi_rx_hs_clk_phy:http://pastebin.com/GW814mxZcsi_rx_hs_lane_phy:http://pastebin.com/UXAMmWeemipi_phy_stats holds reset high for 3 byte_clock cycles before going low. This reset pin is connected to ISERDESE2. I'm not sure about input delay constraints for SERDESs. Hopefully someone can enlighten me. The concerning issue is that my clock frequencies seem to be all over the place. To get an idea I have made a short video:Shouldn't my input clock and byte_clock be a bit more stable than that?

author: daveshah
author_rank: Unknown
timestamp: February 07, 2017, 11:51:51 am
content: At the moment the camera is configured to gate the clock lane inbetween packets which is probably the cause of the the unstable clock frequency. I'm not completely sure but I think the SERDESs might not like an unstable clock. It could also be a signal level issue.The quick test is to disable clock lane gating by setting register 0x4800 from 0x24 to 0x04 and seeing what happens to the clocks.Another thing to try in case there are any timing or signal integrity issues is to reduce the MCLK going into the camera - say by a factor of 2 - to reduce the MIPI clock frequency correspondingly and see what happens to the number of SoTs.

author: malkauns
author_rank: Contributor
timestamp: February 07, 2017, 08:26:56 pm
content: Your suggestion helped! My clocks are now stable:However the SoT's are still not that stable and the average per second seems to change over time (sometimes around 10000/sec, sometimes 2000/sec etc.). I guess this may be more of a register setup issue. I'll do some more debugging.

author: daveshah
author_rank: Unknown
timestamp: February 07, 2017, 09:21:53 pm
content: At some point it might be worth adding some logic to look at the packet headers - 4 bytes following the SoTs. In the CSI spec (google and you will find if you haven't found it already) there's more info.Basically it would be interesting to log whether the ECC is valid and the data type is one that you are expecting.

author: malkauns
author_rank: Contributor
timestamp: February 13, 2017, 06:25:54 am
content: So... I've got to the point where I am detecting simultaneous SoT's on both lanes (about 20000/sec with my current register setup) so I know I'm doing something right. The problem now is that I am not able to generate the expected ECC. I have played with all combinations of register 4801 which sets the bit and byte ordering for the expected ECC generation on the sensor. For each of those combinations I have tried all combinations of bit and byte ordering of the word count (16 bits) and Data ID (8 bits). Nothing seems to get me close to a 100% expected ECC match rate. Right now I am getting a 1% match rate which is probably due to collisions.I expect the lanes to look like this where each element within <> is 8 bits:Code:lane0: <SoT><data_id><word_count(1)>lane1: <SoT><word_count(0)><expected ECC>I pass this data to the ECC function like this and have tried all byte/bit order combinations:Code:<data_id from lane0><word_count(0) from lane1><word_count(1) from lane0>I am using the ECC code from your source(csi_rx_hdr_ecc.vhd)and passing it this data but something is wrong somewhere. Hopefully you (or someone) can shed some light.

author: daveshah
author_rank: Unknown
timestamp: February 13, 2017, 09:20:47 am
content: I think there's a reasonable chance the problem is somewhere in your alignment code - would you be OK posting that?It would also be worthwhile looking to see what values you are getting for data type and packet length (for short packets the latter is actually a 16-bit parameter) - and if they are reasonable. You could do this using the Integrated Logic Analyser for example.

author: malkauns
author_rank: Contributor
timestamp: February 13, 2017, 09:50:45 am
content: My code keeps a 64 bit window of the incoming data on each lane from the camera. I chose 64 bits to account for any slight alignment differences between the lanes. The code shifts everything in this buffer along 8 bits to the left on every byte_clock cycle and writes in the new bits at (7 downto 0):Code:data_0_buf((BYTEBUFLEN-1) downto 8) := data_0_buf((BYTEBUFLEN-1-8) downto 0);data_1_buf((BYTEBUFLEN-1) downto 8) := data_1_buf((BYTEBUFLEN-1-8) downto 0);data_0_buf(7 downto 0) := data_0;data_1_buf(7 downto 0) := data_1;BYTEBUFLEN= 64.data_0anddata_1are the current bytes from the ISERDESE2.I have a function that searches for the last index of the SoT's within those buffers. When I see SoT's on both lanes the following code is triggered:Code:header(31 downto 24) := data_0_buf((data_0_r-1) downto (data_0_r-8)); --Data IDheader(23 downto 16) := data_1_buf((data_1_r-1) downto (data_1_r-8)); --Word Count_0header(15 downto 8) := data_0_buf((data_0_r-1-8) downto (data_0_r-8-8)); --Word Count_1header(7 downto 0) := data_1_buf((data_1_r-1-8) downto (data_1_r-8-8)); --ECCdataid := header(31 downto 24);wordcount(15 downto 8) := header(23 downto 16);wordcount(7 downto 0) := header(15 downto 8);expected_ecc := header(7 downto 0);data_0_randdata_1_rare integers marking the start offset of the SoT on each lane.ECC function:Code:function doecc(data: std_logic_vector(23 downto 0)) return std_logic_vector isvariable ecc : std_logic_vector(7 downto 0);beginecc(7) := '0';ecc(6) := '0';ecc(5) := data(10) xor data(11) xor data(12) xor data(13) xor data(14) xor data(15) xor data(16) xor data(17) xor data(18) xor data(19) xor data(21) xor data(22) xor data(23);ecc(4) := data(4) xor data(5) xor data(6) xor data(7) xor data(8) xor data(9) xor data(16) xor data(17) xor data(18) xor data(19) xor data(20) xor data(22) xor data(23);ecc(3) := data(1) xor data(2) xor data(3) xor data(7) xor data(8) xor data(9) xor data(13) xor data(14) xor data(15) xor data(19) xor data(20) xor data(21) xor data(23);ecc(2) := data(0) xor data(2) xor data(3) xor data(5) xor data(6) xor data(9) xor data(11) xor data(12) xor data(15) xor data(18) xor data(20) xor data(21) xor data(22);ecc(1) := data(0) xor data(1) xor data(3) xor data(4) xor data(6) xor data(8) xor data(10) xor data(12) xor data(14) xor data(17) xor data(20) xor data(21) xor data(22) xor data(23);ecc(0) := data(0) xor data(1) xor data(2) xor data(4) xor data(5) xor data(7) xor data(10) xor data(11) xor data(13) xor data(16) xor data(20) xor data(21) xor data(22) xor data(23);return ecc;end;

author: daveshah
author_rank: Unknown
timestamp: February 13, 2017, 04:22:02 pm
content: If nothing else I think there's an endianness problem going on. The ECC calculating function expects the input in the formCode:<wordcount MSB> <wordcount LSB> <data type>I also presume that your code looking for SoT only triggers once enough data is in the buffer.