Is there any tool to convert hex files to int32? - Page 1
Forum section: electronics / general

author: ali_asadzadeh
author_rank: Super Contributor
timestamp: March 28, 2019, 10:24:21 am
content: Hi,I have captured some data from ADC with my FPGA using UART, the output captured data is like this, I want to know if there is a readily available tool to convert the data which is in hex into int32 so that I can plot the data using matlab.Note that the data is 24bit and I have added FF to separate the data, I can change the data format to 32bit by replacing the FF with 00, also the data is a signed data.

author: Yansi
author_rank: Super Contributor
timestamp: March 28, 2019, 10:25:42 am
content: That is not an HEX (intelHEX) file.Just refactor the text using simple C program, written for purpose. Would get you somewhere within couple of tens of minutes. Parsing this is very easy. (even using directly matlab I'd say).//EDIT: Also please note, that by "just adding FF or 00" in front of a 24bit two's complement (signed) number does not cut the mustard. You must properly sign extend.FF FC B5 EE is correct, but FF 01 C4 6C is not, should be 00 01 C4 6C. Copy the uppermost bit into the preceding byte.

author: ali_asadzadeh
author_rank: Super Contributor
timestamp: March 28, 2019, 10:43:49 am
content: QuoteThanks for the hint, I'm aware of itQuoteI didn't want to write a c program for it, because I think it should be probably something out there that would do the job.

author: Yansi
author_rank: Super Contributor
timestamp: March 28, 2019, 10:57:56 am
content: Yes, there is, for example the matlab to begin withhttps://se.mathworks.com/matlabcentral/answers/312599-how-do-i-parse-this-complex-text-file-with-textscanIt is as easy to do as in C. /* The C being likely the easier way*/

author: newbrain
author_rank: Super Contributor
timestamp: March 28, 2019, 12:17:10 pm
content: If you have a unix shell available (and chances are very high, among W10, Linux and macOS) you can use something as:Code:xxd -r -p dump.txt | od -t u4 -Anon the raw hex data (no header or footer: tail can come to the rescue).This assumes the 24 bit data is big endian, and does not correct the sign and magnitude (I think that is easier in matlab)man xxdman odStill, I would write a short C program...

author: agehall
author_rank: Frequent Contributor
timestamp: March 28, 2019, 01:04:02 pm
content: I'm surprised so many suggest C. Isn't this exactly what your average scripting language is for? Python, PHP, Pike, JavaScript, Perl etc etc.Anyhow, a quick custom piece of code is probably the way to go so that you have easy control over the parsing and the output.

author: NorthGuy
author_rank: Super Contributor
timestamp: March 28, 2019, 01:14:32 pm
content: How are you going to use the converted data? Read it into some sort of program? Then just modify your program to include parsing. It's 10 minutes of work.

author: newbrain
author_rank: Super Contributor
timestamp: March 28, 2019, 01:29:15 pm
content: Quote from: agehall on March 28, 2019, 01:04:02 pmYou are probably right: I was thinking of doing it in PowerShell just for kicks...but then in a 10 minutes break I concocted this (use as a filter on command line):Code:#include <stdio.h>#include <string.h>#define SIZE 1024 /* Maximum allowed line length */char lineBuffer;int main(void){/* Skip header - 3 lines */for (int i = 0; i < 3; i++)fgets(lineBuffer, SIZE, stdin);/* Stop at first short line (or EOF) */while(fgets(lineBuffer, SIZE, stdin) && (strlen(lineBuffer) > 2)){char *s = lineBuffer;unsigned short int l, m, h;/* Parse groups, ignore delimiter value, stop at first problem */while(3 == sscanf(s, "%*X %hX %hX %hX", &h, &m, &l)){/* Assume big endian */int value = (h << 16) + (m << 8) + l;if(value >= 0x800000)value -= 0x1000000; /* Adjust negative numbers */printf("%d\n", value); /* Write output */s += 12; /* Next group */}}return 0;}It's throwaway code, not much error checking or style, HTH though...

author: rstofer
author_rank: Super Contributor
timestamp: March 28, 2019, 02:40:13 pm
content: As mentioned above, there is a simple MATLAB program for doing this. But MATLAB costs money so maybe it can be ported to Octave.Octave has hex2dec() so you can do something like hex2dec(ff) and get back 255. I didn't pursue it further...

author: amyk
author_rank: Super Contributor
timestamp: March 29, 2019, 02:59:26 am
content: Quote from: agehall on March 28, 2019, 01:04:02 pmIf you're working with embedded stuff, chances are that you're familiar enough with C to do it easily.Bit/byte manipulation is notably not very straightforward in scripting languages, because they were designed for higher-level tasks.Here's my 3-minute attempt:Code:#include <stdio.h>main() {int a, b, c, d;while(scanf("%x %x %x %x ", &a, &b, &c, &d) == 4) {a = b & 128 ? 255 : 0;a = (a << 24) | (b << 16) | (c << 8) | d;fwrite(&a, 1, 4, stdout);}}It would take me much longer to search for an existing program to do it.

author: agehall
author_rank: Frequent Contributor
timestamp: March 29, 2019, 06:09:44 am
content: Quote from: amyk on March 29, 2019, 02:59:26 amYou have obviously been using the wrong scripting languages then.Sane ones are pretty much the same as C for those operations.

author: mariush
author_rank: Super Contributor
timestamp: March 29, 2019, 08:03:13 am
content: Here's how you would do it in PHP :In Windows, just download php in a folder, for example C:\Programs\PHP and then open a command prompt in the same folder with your php script and text file and sayc:\programs\php\php.exe script.phpChange file name in the script if you want. Script just dumps each number on one line. You can say php.exe script.php >output.txt to redirect console output to a text file.You can modify the script so that instead of a new line, the script prints a "," or tab ("\t") or whatever character you want.Code:<?php$ENTER="\x0D\x0A";// Windows uses CRLF , Linux and only one character... change this as needed// determine the path from where the script runs , and change all \ slashes to / slashes - Windows can work with either.$folderRoot=str_replace('\\','/',dirname(__FILE__));if (substr($folderRoot,strlen($folderRoot)-1,1)!=='/')$folderRoot.='/';// change the file name here or replace with $argv to read the file name from command line ex php.exe script.php filename.txt -> $argv=filename.txt// though it will only work if files have no spaces in their names$content=file_get_contents($folderRoot.'20190328142916.txt');$lines=explode($ENTER,$content);$v= array();$vindex=0;foreach ($linesas$line) {if (isHexLine($line)==TRUE) {$values=explode(' ',$line);foreach ($valuesas$value) {if ((ctype_xdigit($value)==TRUE) && (strlen($value)==2)) {// hexadecimal , one byte$nr=hexdec($value);$v[$vindex] =$nr;$vindex++;if ($vindex==4) {// read 4 bytes// comment next line if you don't want the first byte to be reset$v =0;$nr= ($v<<24) + ($v<<16) +($v<<8) +$v;echo$nr.$ENTER;$v =0;$v=0;$v=0;$v=0;$vindex=0;}}}}}// function checks if the line of text is composed of only hexadecimal characters or spacefunctionisHexLine($text) {if (strlen($text)==0) returnFALSE;for ($i=0;$i<strlen($text);$i++) {$c=substr($text,$i,1);if (($c==' ') || (ctype_xdigit($c)==TRUE)) {// it's space or hexadecimal char, so continue} else {returnFALSE;// bad char, don't process this line}}returnTRUE;}?>

author: ali_asadzadeh
author_rank: Super Contributor
timestamp: March 29, 2019, 08:13:47 am
content: Wow...really thanks guys, I did not expect that, really appreciate the tips. though I'm a fan of Node and Javascript

author: ali_asadzadeh
author_rank: Super Contributor
timestamp: March 29, 2019, 11:02:16 am
content: So for completeness I have written the Node version tooCode:<?phpconstfs= require('fs');constEOL= require('os').EOL;constcontents=fs.readFileSync('./20190328142916.txt');constreg= /FF\s+(\w{2})\s+(\w{2})\s+(\w{2})/g;let group=reg.exec(contents);let combinedDecimalValues='';while (group) {consthexValue= `${group}${group}${group}`;constparsedValue=parseInt(hexValue,16);let decimalValue=parsedValue;if (parsedValue>0x7FFFFF) {decimalValue=parsedValue-0xFFFFFF-1;}combinedDecimalValues+=decimalValue+EOL;group=reg.exec(contents);}fs.writeFileSync('./output.txt',combinedDecimalValues);mariush how did you highlighted your code?Do we have code highlight in here?

author: magic
author_rank: Super Contributor
timestamp: March 29, 2019, 02:29:42 pm
content: That's how:Code:<?phpconstfs= require('fs');constEOL= require('os').EOL;constcontents=fs.readFileSync('./20190328142916.txt');constreg= /FF\s+(\w{2})\s+(\w{2})\s+(\w{2})/g;let group=reg.exec(contents);let combinedDecimalValues='';while (group) {consthexValue= `${group}${group}${group}`;constparsedValue=parseInt(hexValue,16);let decimalValue=parsedValue;if (parsedValue>0x7FFFFF) {decimalValue=parsedValue-0xFFFFFF-1;}combinedDecimalValues+=decimalValue+EOL;group=reg.exec(contents);}fs.writeFileSync('./output.txt',combinedDecimalValues);In general, you can click Quote to obtain "source code" of other people's posts.

author: Siwastaja
author_rank: Super Contributor
timestamp: March 29, 2019, 03:40:34 pm
content: Quote from: agehall on March 28, 2019, 01:04:02 pmNothing surprising here. It's a 5-minute throwaway job onanyhalf-decent language. It's approx 10-20 lines on any language, Python or C.The result (how easy and fast it is) depends entirely on the language skills and the current mindset of the programmer. If they have written C for an embedded project in last years, trying to do it in Python because it's somehow "easier" could end up being a disaster in time usage.Really almost any language will do. Trying to find some academical "optimally easy" solution doesn't make a lot of sense if it isn't easy in your specific case.When you have a round peg and a round hole and need to apply just a little bit of extra force to get it in, it doesn't matter much which kind of hammer you end up using, most will do, and you'll use what you have already in your hand, as long as it isn't clearly wrong. Shopping for an optimum type of hammer would make zero sense.For a beginner, Python - as well as many other scripting languages - are good choices,ifthey don't know any language yet, and need to start by learning the language and the tools.