you have raw bytes: how to understand the used lossless algorithm? - Page 1
Forum section: electronics / general

author: DiTBho
author_rank: Super Contributor
timestamp: June 14, 2022, 04:32:05 pm
content: so, yesterday I found a source shooting two packets of 500K raw byte each ever 208 secondsI am 100% sure1) they are not encrypted2) they are lossless images3) 24bit color, 3 byte per pixelwhich is good, but ...how to understand the used lossless algorithm?

author: AndyBeez
author_rank: Frequent Contributor
timestamp: June 14, 2022, 04:52:31 pm
content: Any more clues? A RAW image stream off a CCD is lossless. So too is Windows BMP. In what context is the image data? Is it a TS transport stream?

author: DiTBho
author_rank: Super Contributor
timestamp: June 14, 2022, 05:07:51 pm
content: Quote from: AndyBeez on June 14, 2022, 04:52:31 pmimage data

author: DiTBho
author_rank: Super Contributor
timestamp: June 14, 2022, 05:11:16 pm
content: Code:0x00, 0x00, 0x00, 0x00, 0x00, 0x01, 0x00, 0xFF, 0x00, 0x01, 0x00,0x01, 0x01, 0x00, 0x00, 0xFF, 0xFF, 0x00, 0x00, 0x00, 0x05, 0x01,0x01, 0x03, 0xFF, 0x00, 0x00, 0x03, 0x01, 0x00, 0x01, 0x01, 0x00,0xFF, 0x00, 0x00, 0x01, 0xFF, 0x01, 0xFF, 0x01, 0x00, 0x00, 0x02,0xFF, 0x00, 0x00, 0x00, 0x01, 0xFF, 0xFF, 0x00, 0x00, 0x00, 0x00,0x01, 0x01, 0x00, 0x00, 0x01, 0x00, 0x00, 0x00, 0xFF, 0x01, 0x00,0xFF, 0xFF, 0x01, 0x01, 0x01, 0x00, 0xFF, 0xFF, 0x00, 0x00, 0x03,0x00, 0x00, 0x00, 0x01, 0xFF, 0xFE, 0x00, 0x00, 0x01, 0xFF, 0x01,0x00, 0x01, 0x00, 0xFF, 0xFF, 0x01, 0x00, 0x00, 0x02, 0xFF, 0x01,0xFF, 0x00, 0x01, 0x00, 0x01, 0x00, 0x00, 0x01, 0xFF, 0xFF, 0x00,0xFE, 0x01, 0x00, 0x00, 0x00, 0x01, 0x00, 0x01, 0x00, 0xFF, 0xFF,0x00, 0x00, 0x00, 0x00, 0x01, 0x00, 0x01, 0x00, 0xFF, 0xFF, 0x00,0x01, 0x00, 0xFF, 0x01, 0xFE, 0x00, 0xFF, 0x00, 0x00, 0x02, 0x01,0x00, 0x00, 0x00, 0xFF, 0xFF, 0x01, 0x00, 0x00, 0x05, 0xFF, 0xFF,0x00, 0x00, 0x00, 0x00, 0x01, 0x00, 0x00, 0x00, 0xFF, 0x00, 0xFF,0x01, 0x01, 0x00, 0x00, 0x01, 0x00, 0xFF, 0x00, 0x01, 0x01, 0x00,0x00, 0x00, 0x01, 0xFF, 0x01, 0x00, 0x01, 0x01, 0x00, 0x00, 0xFF,0x01, 0xFF, 0x00, 0x00, 0x01, 0x01, 0x01, 0x01, 0x00, 0x00, 0x00,0x01, 0x00, 0x00, 0x00, 0x01, 0x00, 0xFF, 0x01, 0x00, 0x00, 0x00,0xFF, 0x00, 0x00, 0x00, 0x01, 0xFF, 0x00, 0x00, 0x00, 0xFF, 0x00,0x00, 0x00, 0xFE, 0x01, 0x00, 0xFF, 0x00, 0x01, 0x00, 0xFF, 0x00,0x01, 0x00, 0x01, 0x00, 0x01, 0x01, 0x00, 0x00, 0x00, 0x02, 0x01,0x00, 0xFF, 0x00, 0x00, 0x01, 0xFF, 0xFF, 0x00, 0x00, 0xFF, 0x00,0x00, 0x05, 0x02, 0x01, 0xFF, 0x01, 0xFF, 0xFF, 0x00, 0x00, 0x00,0x01, 0x01, 0x01, 0x00, 0xFF, 0x00, 0x00, 0x00, 0xFF, 0x00, 0x01,0x01, 0x00, 0xFF, 0xFF, 0x01, 0x00, 0x00, 0x00, 0x01, 0x00, 0x00,0x00, 0xFF, 0xFE, 0x00, 0x00, 0x01, 0xFF, 0x00, 0xFF, 0x00, 0x00,0x00, 0xFF, 0x00, 0x00, 0x02, 0xFF, 0x00, 0x01, 0x01, 0x00, 0x00,0x00, 0x03, 0x01, 0xFF, 0x00, 0x01, 0xFF, 0x01, 0x00, 0xFF, 0x01,0x00, 0x00, 0x00, 0x01, 0x00, 0xFF, 0x00, 0xFF, 0x00, 0x00, 0x04,...This is how it looks

author: AndyBeez
author_rank: Frequent Contributor
timestamp: June 14, 2022, 05:43:19 pm
content: Strange indeed. All I can 'suspect' is the 0xFF is some kind of control byte that terminates a data group.Syntax wise: { ... 0xFF }...But how it maps to pixels, I have 0.0 idea?

author: DiTBho
author_rank: Super Contributor
timestamp: June 14, 2022, 06:53:22 pm
content: umm, it seems the compression is done by a small FPGA, something like Xilinx Spartan3 400.So, the next question is: which are the best image lossless compression algorithms that you can implement in a small FPGA?

author: DiTBho
author_rank: Super Contributor
timestamp: June 14, 2022, 06:58:05 pm
content: or, which are the most widely used among lossless data compression algorithms for hardware implementation?(then I will try to apply their decoders, and see ... it something matches)

author: mariush
author_rank: Super Contributor
timestamp: June 14, 2022, 07:32:11 pm
content: Can you upload the whole 500 KB or whatever the amount is sent?There may be a color palette at the end, or something like that.Curious if it's always 0xFF ... was thinking that byte is used to signify something as in if bit is 1 then it's a reference to a color palette entry, 0 means something else (actual color definition or something like that, like new color information, and add to the color palette)Too many 0x00 in series to be actual compression imho... at most could be some RLE scheme

author: DiTBho
author_rank: Super Contributor
timestamp: June 14, 2022, 08:32:20 pm
content: attached a binary image with two consecutive packetsimage.tar(324 kB - downloaded 86 times.)

author: DiTBho
author_rank: Super Contributor
timestamp: June 14, 2022, 11:15:56 pm
content: I wonder ... MJPEG (Motion JPEG)?Implementing MJPEG in a small fpga le 3S400 should be doable

author: DiTBho
author_rank: Super Contributor
timestamp: June 14, 2022, 11:25:09 pm
content: Yesterday I hacked the Kodak PixCam used in Palm m100 era.Nothing special, it's a 320x240x24bit-color(1) pixel camera that shoots over a serial line, but the used format was unusual, because it adds a marker of 4 byte every 14440 byte.Stupid, but easy to get.Unfortunately it's not the case here(1) true false colors, the first byte is always 0x0, so it really only uses 4 bits but even if it pretends to store every R, G, B channel in one byte. I tried every single official software, no dice, always the same.

author: golden_labels
author_rank: Super Contributor
timestamp: June 15, 2022, 12:00:31 am
content: The data has low symbol-to-symbol entropy: you can tell by it being easily compressible by 75% with any modern algorithm. That excludes MJPEG, as Huffman coding would prevent that. Same goes for any reasonable compression, obfuscation or encryption algorithm.If that stores data as separate pixels, you may pick an arbitrary fragment and try displaying it using different row lengths and bits per pixel. If you see some patterns, adjusting those two parameters will at some point produce the desired image and that will tell you what is the actual geometry.

author: brucehoult
author_rank: Super Contributor
timestamp: June 15, 2022, 12:25:38 am
content: Quote from: DiTBho on June 14, 2022, 05:11:16 pmIf that's compressed it's pretty badly compressed! gzip cuts that down from 330 bytes to 150. Even lz4 reduces it to 227. They would both do much better on the full file if it continues like that.

author: DavidAlfa
author_rank: Super Contributor
timestamp: June 15, 2022, 01:00:31 am
content: If the receiver knows the resolution beforehand, it2 xould4 be just raw RGB data.If the packets are always the same size, I'd discard any compression.Do you know the resolution of the device?Attach a full packet, so we can play with it...

author: Mechatrommer
author_rank: Super Contributor
timestamp: June 15, 2022, 01:46:00 am
content: Quote from: DiTBho on June 14, 2022, 11:15:56 pmdefinitely no. RLE at best... or just a custom format. probably MSB (sign bit) (or LSB?) may tell something.

author: SiliconWizard
author_rank: Super Contributor
timestamp: June 15, 2022, 02:03:58 am
content: It's very likely a form of RLE. You'd need to analyze the data for given known frames with simple content to be able to figure it out in a reasonable amount of time.

author: DiTBho
author_rank: Super Contributor
timestamp: June 15, 2022, 11:31:20 am
content: I am 100% sure that the source is a 640x480 image with 1 byte for each color, so 24 bit colorI will try to capture a compressed frame with the highest compression ratio. I mean, I noticed that if I try to congest the link then there are two packets of 50-40 Kbytes in size instead of two packets of 400-500 KbytesThis should probably mean that the compression ratio is "adjustable" from "low" to "extreme"

author: dietert1
author_rank: Super Contributor
timestamp: June 15, 2022, 01:08:12 pm
content: Somehow this reminds me of a 2009 project, description here:http://cadt.de/notes/DesktopVI.pdf. Image compression happened on a Digilent SpartanII board. It had lossless compression by about a factor 100 in realtime with a throttle algorithm. The idea was to get a live color display from a scope with a grayscale screen. The pdf includes a link to a youtube video that is still available..These things depend a lot on the context. E.g. in the case of the Lecroy a fixed palette and an almost empty screen. Is the image available that belongs to the data packets posted above?Regards, Dieter

author: emece67
author_rank: Frequent Contributor
timestamp: June 15, 2022, 01:30:02 pm
content: .

author: mikeselectricstuff
author_rank: Super Contributor
timestamp: June 15, 2022, 01:51:04 pm
content: If possible, looking at how the compressed image size varies with content may give some clues.e.g. show it limited range of colours, large/small amoints of detail in X & Y, mono vs. colour images etc.

author: m k
author_rank: Super Contributor
timestamp: June 15, 2022, 04:08:35 pm
content: 1028 bytes wide editor would be nice.(cut 1st 0x200 off)E,counter of sorts is indicating a base color/line.

author: DiTBho
author_rank: Super Contributor
timestamp: June 16, 2022, 09:24:35 am
content: I finally captured a packet containing an image compressed with level=5, which should be "extreme" compression.image.tar(199.85 kB - downloaded 70 times.)

author: DiTBho
author_rank: Super Contributor
timestamp: June 16, 2022, 10:08:45 am
content: Quote from: SiliconWizard on June 15, 2022, 02:03:58 amWhat if one day we will receive a data-packet from a radio telescope capturing things from outer-space?You don't know the encoding algorithm, you don't know the compressing algorithm, ...Code:________________| |A L I E N | |unknown message ------> x | blackbox | y --------> message received| y = h(x) || ||________________|Code:________________| |H U M A N | |unknown message ------> x | blackbox | y --------> message received| y = f(x) || ||________________|Code:________________| |H U M A N | |known message --------> x | blackbox | y --------> message received| y = f(x) || ||________________|In this case we are lucky because I can force a known-image into the black-box and capture the outputimage.tar(15.12 kB - downloaded 82 times.)

author: Mechatrommer
author_rank: Super Contributor
timestamp: June 16, 2022, 11:00:57 am
content: Quote from: DiTBho on June 16, 2022, 10:08:45 amyou call linguists, cryptologists, expert in encoding schemes, scientists etc etc... decoding unknown and complex data communication is not a one man job. on earth alone, several disciplines are needed to decode just for say, Egyptian hieroglyphs.. btw, your level 5 lossless compression looks like not a compresion at all (see attached).. maybe next test is to send the blackbox a noncompressible (random noise) image and ask it to do maximum compression regardless...

author: DiTBho
author_rank: Super Contributor
timestamp: June 16, 2022, 11:43:42 am
content: Quote from: Mechatrommer on June 16, 2022, 11:00:57 amHollywood makes first contact with aliens a breeze, but it's not.I'm trying to train myself to decode everything I can access just in case ... and it's so damn hard, as you can see here.Four bloody weeks just to sniff the USB protocol, figure out where frames start and end, and still don't understand how they are compressed.I have a purpose for that device, it's not just a game, but it's also a challenge to understand how complex certain things are in reality.I mean, I love comics, I've read tons of comics and watched hundreds of movies, and in a Superman movie you see Lex Luthor able to handle how to decode an alien algorithm to unlock the door of an alien spaceship and immediately access the door. . entire Cryptorian database to create the worst villain ever, Doomsday, whose birth has only one purpose: Superman's death.If you watch the movie, in less than 48 hoursLex Luthorwas also able to decode a super complex Cryptorian technology of a bio-weapon classified as "only generals can access it", and he easily bypassed the security level and got it. . .Insanely bright! Is he a super-ultra-sapiens human being? the evolution of human beings, hence more intelligent, clever and smart than Einstein?Crazy Hollywood! This part sucks because it's too far from realityQuote from: Mechatrommer on June 16, 2022, 11:00:57 amNo, they are two different images. The one with the finger is the only one frame I can capture with a known input image.The skull one comes from a different source, which I cannot control.But, if it's not compressed, then it's interesting because it means that sometimes it shoots an uncompressed frameSo, I have to look at the control frame to see if there is a special command to issue a not compressed shot.I have to improve the C source that tries to capture frames (based on libusb, on Linux). It's not easy, there are a lot of control packets in the middle.But images are only transported on bulk-frames (up to 500Kbyte) and I cannot simply ignore everything else because I have to check the compression level, which usually precedes the bulk frame.Ctrl frame, it sometimes report the compression level, a number from 1 to 5bulk frame, 1 of 4, sometimes it's 1 of 2, sometimes it's 1 of 1, it depends on the compression level and on the image size

author: DiTBho
author_rank: Super Contributor
timestamp: June 16, 2022, 11:59:46 am
content: Code:. _____hidden source --------| | __________| | | || mux |-------- x | blackbox | y ------| | | y = f(x) |external source --------|_____| |__________|^|compress levelvalue {1 .. 5}ummm, it looks like I set the mux incorrectly, so the blackbox ignored the external source and processed the hidden source but, this time, no compression. I have to recreate the packet sequence for this, because it is very interesting and it is the first time that it emits something uncompressed.Anyway, I issued an hardware reset, set the mux to the external source, and captured the compressed packet.This time it should be right.image.tar(15.12 kB - downloaded 80 times.)

author: DiTBho
author_rank: Super Contributor
timestamp: June 16, 2022, 12:05:19 pm
content: @Mechatrommerwhich program are you using to decode pictures?here I have implemented a very wild program for x11Code:for (y = 0; y < ysize; y++){for (x = 0; x < xsize; x++){if (i < data_size){R = data[i + 0];G = data[i + 1];B = data[i + 2];}else{R = 0;G = 0;B = 0;}x11_point_color(R, G, B);x11_point_draw(x, y);i += 3;}}

author: golden_labels
author_rank: Super Contributor
timestamp: June 16, 2022, 12:10:49 pm
content: Wait, do you have something to see a decoded image? If yes, just introduce modifications. Flip a bit or increment some octet and see what happens.As for the theoretical part: with zero knowledge about the content of the message and the encoding mechanism, you can never decode it. That is because you have no way of discerning valid decodings. Which is why one-time-pad is provably unbreakable encryption even if you could bruteforce all possible keys. Same goes for single blocks of many block ciphers.IMO communicating with extraterrestial species would be much worse than just trouble with encoding. If both sides had the same goals while designing the communication methods, both would probably end up with something recognizable to the other side. The true obstacle may be of different nature. No concepts shared. Humans love to model ETs as “just an exotic civilization”, but in reality they may have been shaped to use concepts completely unfamiliar to humans. Make a simple thought experiment that doesn’t even require involving other planets. How would you talk with ants or trees, given you had a chance to do so? You may argue they are not complex enough and sure I do not ask about having a debate on philosophy with a nearby ant hill. But if you consider them horribly dense and only want to communicate simple things… you still can’t. Their and your pereception of everything is just too different to share any kind of information.

author: Mechatrommer
author_rank: Super Contributor
timestamp: June 16, 2022, 12:25:38 pm
content: Quote from: DiTBho on June 16, 2022, 11:43:42 ambecause most aliens there speak english, and with decades of mixed culture, you can quickly tell if raising a hand is a hi or an insultwatching too much "Marvel" grade movies is not good for your cryptanalyst's health, even experts will stuggle on others' creations in reality... if you want to watch movies instead of learning most decoding (encryption and compression) schemes in real life and try to invent one yourself, may i suggest history movies such as Midway(2019) or The Imitation Game(2014). cheersQuote from: DiTBho on June 16, 2022, 12:05:19 pmits my diy tool i made a while ago to extract plain image resources from some apps, it supports simple pixel and width padding and non 8-bit aligned data. also can be used to see data compressibility or randomness (information's entropy) in visual color form...

author: DiTBho
author_rank: Super Contributor
timestamp: June 17, 2022, 09:59:59 pm
content: what do you think, guys, about the last binary image (compression level 5) I uploaded?I captured a document that mentions "AVI", which somehow gives more clues.However AVI is only a container, which can use:- MPEG-1- MPEG-2- MPEG-4- WMV- RealVideo- Theora- FlashOMG

author: Mechatrommer
author_rank: Super Contributor
timestamp: June 17, 2022, 11:37:53 pm
content: Quote from: DiTBho on June 17, 2022, 09:59:59 pmthats life. avi is an old format that has seen many new compression schemes coming in... backward compatibility has to be preserved.

author: DiTBho
author_rank: Super Contributor
timestamp: June 25, 2022, 07:29:10 pm
content: I am trying with lz77&C compression-algorithms

author: Mechatrommer
author_rank: Super Contributor
timestamp: June 26, 2022, 06:00:47 am
content: Quote from: DiTBho on June 25, 2022, 07:29:10 pmi suggest start in chronological order, such as RLE and then huffman encoding and to more advanced dictionary based with tree traversing etc etc... so you can understand why things happened as it happened. its a long story cant easily being told in a single forum. good luck!

author: DiTBho
author_rank: Super Contributor
timestamp: June 26, 2022, 09:09:03 am
content: Quote from: Mechatrommer on June 26, 2022, 06:00:47 ameheh, and it will be a solid-rock learning experience this way!Good idea! Thanks