Worn out flash memory chip (update: maybe not)? - Page 1
Forum section: electronics / repair

author: TheMG
author_rank: Frequent Contributor
timestamp: October 07, 2019, 05:13:30 pm
content: Have this network management card for a UPS which has been having some issues. The initial problem was it would randomly reset all the time, every few minutes. I attempted a defaults reset to see if that would alleviate the problem, but upon trying to reconfigure the card, every time I try to change the IP address, it crashes and when it comes back up it's back to the default IP.So the next step I took was to operate the card on the bench from an external power supply, and check the onboard regulators (3.3V) for proper voltage and ripple. Everything looks ok there. While doing this, I noticed the card was not randomly resetting like it was doing when installed inside the UPS.Thinking maybe the firmware is corrupt somehow, I attempted to reload the firmware... no luck, hangs in the middle of the firmware upload process.My hypothesis is that the flash memory chip is simply worn out from too many write cycles, and any time it tries to write to a bad memory block it causes the card to crash and reboot.The flash chip is a 39VF800A in a 48-TSOP package.In order to replace the memory chip, I would need a reader/programmer to make a copy of the contents of the memory, and write the contents to the new memory before soldering it to the board. I've been putting off getting a programmer for a while and this is as good an excuse as any to finally buy one. Can anyone suggest a decent universal chip programmer capable of handling this chip (as well as other common memory chips and micro-controllers)?Also as a side note, can anyone identify the CPU on this thing? AC1101 brings up absolutely nothing in any datasheet searches. I can't even identify the manufacturer, as I don't recognize the symbol.

author: sokoloff
author_rank: Super Contributor
timestamp: October 07, 2019, 07:09:35 pm
content: For $17 to $25, you can buy another one.Given that, I wouldn't spend any time on it.https://www.ebay.com/sch/i.html?_nkw=ConnectUPS-X&_sacat=0&_sop=15

author: TheMG
author_rank: Frequent Contributor
timestamp: October 07, 2019, 07:47:18 pm
content: I realize that. However, I'm in Canada and most of those for sale are in the USA. With the exchange rate, shipping fees, etc, that's $90+ by the time it's all said and done.That, and what's to say it's not going to soon suffer the same fate, since most of these for sale have probably been pulled out of units that were in service for many years, and thus would have significant write cycles on the flash already.Anyways, I will need a programmer for some upcoming projects anyways so I wouldn't be buying it strictly for this one repair. Plus I like the challenge of trying to repair things if I can, rather than throwing away.

author: picburner
author_rank: Frequent Contributor
timestamp: October 07, 2019, 08:05:27 pm
content: The TL866II eprom programmer supports this type of memory, you will also need its TSOP48 - DIP40 adapter of course.

author: TheSteve
author_rank: Unknown
timestamp: October 07, 2019, 08:42:57 pm
content: I would get a TL866II Plus with the various adapters - they sell it all as a single kit.Although I am more suspect of ram then flash.Unless you're near Langley BC and then I'm happy to read the old one and write a new one for you.

author: TheMG
author_rank: Frequent Contributor
timestamp: October 07, 2019, 11:00:43 pm
content: Quote from: edavid on October 07, 2019, 08:53:47 pmThe battery is only for the RTC. Logs and configuration are retained even when the battery is removed for an extended period of time. There is an RTC chip on the main board and it's the only thing the battery feeds power to.Quote from: TheSteve on October 07, 2019, 08:42:57 pm$68 shipped from China, wow! Almost seems too good to be true, but the reviews seem to be pretty good. Purchased one along with the appropriate adapters. Will be useful for programming PIC micros in the future too, instead of buying more expensive ones with a pre-loaded bootloader. Also have a few PICs where I've "bricked" the bootloader so it'll be useful for that too.Quote from: TheSteve on October 07, 2019, 08:42:57 pmYellowknife...

author: negativ3
author_rank: Regular Contributor
timestamp: October 08, 2019, 10:33:33 am
content: In Canada...https://www.ebay.com/itm/POWERWARE-CONNECTUPS-X-WEB-SNMP-xHUB-CARD-5071-00010-A5-price-reduced/183710118030?epid=1869348947&hash=item2ac5f9f08e:g:amYAAOSwcyVceWzs

author: TheMG
author_rank: Frequent Contributor
timestamp: October 30, 2019, 02:09:28 am
content: Well I got the chip programmer, removed the flash chip from the PCB, and read its contents.It doesn't look like the flash memory is to blame, after all, as I've written all FF and all 00 to it and that seems to be fine.But the fact remains, the device is still bricked from the failed firmware upload with no way of recovery besides trying to flash the memory directly.Interestingly, comparing the firmware binary obtained from the manufacturer's website with the contents of the flash memory, it almost looks like the firmware flashing process started to write the firmware near the bottom of the memory, and then "wrapped around" back to the top and kept going from there. Now, maybe this is by design, but seems odd to me especially since the first part of the firmware binary appears to be some type of BIOS. Why would BIOS code be so far down in the memory and not at the very top? Seems suspect.Interestingly, there is in this BIOS what seems to be some diagnostic functions for testing flash, RAM, etc. I could never figure out how to access these "hidden" functions.So there's a couple of things I need to figure out:1 - How to correctly write the firmware to the flash memory. Tempted to try just write the binary firmware file to it sequentially starting at location 00000000 and see what happens.2 - Test the RAM chips somehow, or just replace them.I may be in way over my head on this one, as I'm definitely not much of a programmer, especially since this is a completely unknown CPU with no documentation to be found anywhere.Oh, and, for the record, I did find another one of these cards (newer hardware revision actually) new-in-box for a decent price. But I'd still like to attempt to fix this other one, if I can, if for nothing other than the experience.Binaries attached. The one named "PW338.bin" is the firmware flash file as available on manufacturer's website, the other one is what I've extracted from the actual flash chip.

author: fzabkar
author_rank: Super Contributor
timestamp: October 30, 2019, 06:36:12 am
content: AFAICT the firmware consists of 4 modules:PW338.binCode:offset size description------ ----- ---------------------------------------------------------0 200 0x200 byte header for firmware download (checksum = 0x00)200 f000 BOOT Generic - 0x100 byte header + codef200 1000 BOOT Generic - 0x100 byte header + boot code10200 4b3d0 FILE Ingrasys USHA - 0x100 byte header + code5b5d0 4f800 MAIN ConnectUPS - 0x100 byte header + codeaadcf end of firmware payloadflash_contents.binCode:offset size description------ ----- ---------------------------------------------------------0 a0000 FILE Ingrasys USHA - 0x100 byte header + codea0000 50000 MAIN ConnectUPS - 0x100 byte header + codef0000 f000 BOOT Generic - 0x100 byte header + codeff000 1000 BOOT Generic - 0x100 byte header + boot codefffff end of flashThe firmware payload file (PW338.bin) has a 0x200-byte header which appears to define the destination location and size of each component, but I haven't been able to determine how this is done. This would explain why the modules appear to be out of order. There is a single checksum byte at the end of the header (offset 0x1FF, value 0xB8). This byte is chosen so that the sum of all 0x200 bytes in the header is 0x00.The BOOT block appears to be in the right place for an Intel x86 processor architecture. An Intel CPU begins executing at FFFF:0.Code:Offset(h) 00 04 08 0C000FFFF0 EA000100 FFFFFFFF 30362F32 322F3032 ........06/22/02If you disassemble the code at this point, it executes a JMP instruction which sends it to the first instruction after the 0x100-byte header of the "BOOT Generic" block (FF00:0100).Code:C:\>debug-e 1001262:0100 EA.ea 00.00 01.01 42.00 A0.ff-u 1001262:0100 EA000100FF JMP FF00:0100This JMP behaviour is consistent with PC BIOS.If we now extract each of the 4 components from the flash.bin dump, using the size in the payload, we find that the firmware update aborted part way through the final module, "MAIN ConnectUPS", at offset 0xE8000 in the flash. All other modules match up (apart from a mysterious 4-byte parameter). ISTM that you just need to patch the flash by adding the rest of the "MAIN ConnectUPS" module at 0xE8000. I can do this for you if you can't understand what I'm suggesting.The only query I have is in regard to the meaning of those mysterious 4 bytes in the headers of each of the 4 modules.For example ...PW338.binCode:Offset(h) 00 04 08 0C0005B5D0 5A3F0000 00C02603 00000600 00000A00 Z?..............0005B5E0 08000110 FFFFFFFF 4D41494E 00000000 ........MAIN....^^^^^^^^0005B5F0 436F6E6E 65637455 50530000 00000000 ConnectUPS......flash_contents.binCode:Offset(h) 00 04 08 0C000A0000 5A3F0000 00C02603 00000600 00000A00 Z?..............000A0010 08000110 2EC48C9E 4D41494E 00000000 ........MAIN....^^^^^^^^000A0020 436F6E6E 65637455 50530000 00000000 ConnectUPS......

author: amyk
author_rank: Super Contributor
timestamp: October 30, 2019, 12:21:50 pm
content: Great work, fzabkar! I decided to follow the jump over to FF100 and it looks like valid x86 code to me --- probably 8088/8086-level, since I didn't see any newer instructions. It starts off doing what looks like a self-test of the CPU, does some I/O, then goes to either FB300 where it copies some code into probably RAM and then goes to 1000:0200, or to F0100 with more CPU testing and I/O, maybe a memory test, and a lot more things happen.Looking at the firmware image, I see what looks like .class filenames and reference to .jar --- which could mean this system runs Java

author: fzabkar
author_rank: Super Contributor
timestamp: October 30, 2019, 08:32:03 pm
content: I have extracted the components from PW338.bin and flash_contents.bin:http://www.users.on.net/~fzabkar/temp/UPS_firmware_components.7zThe PW338_MAIN.bin component has a 0x100 byte header, followed by a JMP instruction which jumps over a table of modules and lands on a LOADER module at A042:0100 (= 0xA0520). Note that this MAIN module has an absolute offset of 0xA0000, so the offsets in the following dumps should be viewed as being relative to this base address.Code:Offset(h) 00 02 04 06 08 0A 0C 0E00000100 EA00 0142 A000 0000 0000 0000 0000 0000Code:-u 1001262:0100 EA000142A0 JMP A042:0100Code:Offset(h) 00 02 04 06 08 0A 0C 0E00000520 EB0A 0100 4C4F 4144 4552 0000 6800 20E8 ....LOADER......Here is the table:Code:Offset(h) 00 02 04 06 08 0A 0C 0E00000130 53A1 0000 0000 0000 5061 7373 5468 7275 ........PassThru^^^^ ^^^^^^^^^^^^^^^^^^^absolute segment address module name00000140 5C0B 0000 2A0A 0000 0100 0000 0000 0000 ................00000150 ECA2 0000 0000 0000 554C 5A00 0000 0000 ........ULZ.....00000160 4516 7F00 7114 CC0D 0100 0100 0000 0000 ................00000170 A2A3 0000 0000 0000 5354 5249 4E47 5800 ........STRINGX.00000180 7D78 1330 F173 8B42 0200 0100 0000 0000 ................00000190 7FA4 0000 0000 0000 5359 5354 454D 0000 ........SYSTEM..000001A0 6528 BD58 B125 8619 0100 0100 0000 0000 ................000001B0 A8A8 0000 0000 0000 5345 5256 4943 4500 ........SERVICE.000001C0 EF06 6502 1E05 B803 0100 0100 0000 0000 ................000001D0 41AA 0000 0000 0000 4D49 424D 4752 0000 ........MIBMGR..000001E0 F349 AD08 5F3F 152D 0100 0900 0000 0000 ................000001F0 7DAA 0000 0000 0000 454D 4400 0000 0000 ........EMD.....00000200 7B15 9D05 7D12 4D0A 0100 0100 0000 0000 ................00000210 4FAD 0000 0000 0000 5550 534D 4752 0000 ........UPSMGR..00000220 B360 9D74 D355 D635 0100 0100 0000 0000 ................00000230 F4AD 0000 0000 0000 5445 4C4E 4554 0000 ........TELNET..00000240 7D59 D752 E656 DC39 0100 0100 0000 0000 ................00000250 52B1 0000 0000 0000 5443 5049 5000 0000 ........TCPIP...00000260 BB2B 7F28 F228 7B1A 0100 0900 0000 0000 ................00000270 F0B4 0000 0000 0000 534E 4D50 0000 0000 ........SNMP....00000280 ED3A 7F83 0B36 BA22 0100 0100 0000 0000 ................00000290 98B6 0000 0000 0000 4854 5450 0000 0000 ........HTTP....000002A0 9D24 B216 0523 7816 0100 0900 0000 0000 ................000002B0 C4B8 0000 0000 0000 4448 4350 0000 0000 ........DHCP....000002C0 9B2A 8129 7125 741B 0200 0100 0000 0000 ................000002D0 2CBA 0000 0000 0000 5546 5450 5300 0000 ........UFTPS...000002E0 D340 8311 543B 2423 0100 0900 0000 0000 ................000002F0 E4BB 0000 0000 0000 534D 5450 0000 0000 ........SMTP....00000300 8354 5124 7836 FD2D 0100 0900 0000 0000 ................00000310 17BE 0000 0000 0000 504F 5033 0000 0000 ........POP3....00000320 DFA2 FD41 5F8C 3E5A 0200 0100 0000 0000 ................00000330 F7C0 0000 0000 0000 554E 4D50 5300 0000 ........UNMPS...00000340 CD1A A507 9212 680D 0100 0900 0000 0000 ................00000350 9BC6 0000 0000 0000 4E54 5000 0000 0000 ........NTP.....00000360 B375 DB04 5C2C 8D37 0100 0100 0000 0000 ................00000370 72C7 0000 0000 0000 4C4F 474D 4752 0000 ........LOGMGR..00000380 9D3C 0500 E202 E40B 0100 0300 0000 0000 ................00000390 EBCA 0000 0000 0000 4D49 4200 0000 0000 ........MIB.....000003A0 7B3B 0000 8802 5E1A 0100 0300 0000 0000 ................000003B0 AACB 0000 0000 0000 4D45 4E55 0000 0000 ........MENU....000003C0 8921 AF1A 061D BE14 0100 0100 0000 0000 ................000003D0 50CD 0000 0000 0000 7570 7354 7261 7000 ........upsTrap.000003E0 BC8F D62C B85D BE28 0100 0900 0000 0000 ................000003F0 9CCE 0000 0000 0000 4352 5950 544F 0000 ........CRYPTO..00000400 1A96 623F 2F89 5A4B 0100 0900 0000 0000 ................00000410 28D1 0000 0000 0000 5253 4100 0000 0000 ........RSA.....00000420 B121 D52F 431A 9B15 0100 0900 0000 0000 ................00000430 DED5 0000 0000 0000 5353 4800 0000 0000 ........SSH.....00000440 45F5 C3A7 63C0 997C 0100 0900 0000 0000 ................00000450 38D7 0000 0000 0000 4341 0000 0000 0000 ........CA......00000460 812A 69AC 7628 2E18 0100 0900 0000 0000 ................00000470 02DF 0000 0000 0000 4854 5450 0000 0000 ........HTTP....00000480 070A 0901 B708 7D05 0100 0300 0000 0000 ................00000490 85E0 0000 0000 0000 4445 5445 4354 0000 ........DETECT..000004A0 F16F 5F2F 6564 4645 0100 0500 0000 0000 ................000004B0 DDE0 0000 0000 0000 5550 4D00 0000 0000 ........UPM.....000004C0 6359 0B11 3649 0833 0100 0500 0000 0000 ................000004D0 32E5 0000 0000 0000 5550 4D00 0000 0000 ........UPM.....000004E0 55CC F12E 4C95 3A6F 0100 0100 0000 0000 ................000004F0 63E8 0000 0000 0000 5550 5344 4400 0000 ........UPSDD...Each module in the table is defined by a 32-byte record. This record includes the absolute segment address and the module name.For example, the "PassThru" module is located at absolute address 0xA153:0 (0xA1530). This is consistent with its location in the flash, which confirms that the firmware is being updated at the correct addresses.This is the header of the payload file:Code:Offset(h) 00 04 08 0C 10 14 18 1C00000000 5553 4841 01010002 03002000 08000110 31313134 32303132 2D19EE9F FFFFFFFF00000020 4000 D8BD 80D00200 00000000 00000100 00000F00 00B00904 00000100 00000000 BOOT 1 & 200000040 6000 AE81 00D00200 00000100 D0B30400 00000000 00F00100 D0B30400 00000000 FILE00000060 FFFF A73C 00D00200 D0B30500 00F80400 00000A00 00C02603 00F80400 00000000 MAIN^^^^ **** ^^^^^^^^ -------- ^^^^^^^^ --------| CRC-16 CCITT | size flash address sizeoffset to next |record relative offset in payload.......000001E0 0000 0000 00000000 00000000 00000000 00000000 00000000 00000000 000000B8checksum ^^It would appear that the two BOOT components have been lumped together.I don't understand all the parameters for each component, but it is now clear how the updater knows where to find them and where to write them, and how to verify their integrity.Clearly, all the data are little-endian.The CRC-16 CCITT checksums can be confirmed with HxD (a freeware hex editor):https://mh-nexus.de/en/hxd/

author: TheMG
author_rank: Frequent Contributor
timestamp: October 31, 2019, 01:21:48 am
content: Wow!!! I'm going to need a little bit of time to absorb all this, if I understand the overall picture, the data is in the right place, but one of the firmware "modules" is incomplete due to the firmware upload crapping out mid-process.How did you find all this anyways? Either you have some magical software and/or some mad programming skills. Either way, I'm impressed. Much of the contents of the firmware might as well be a foreign language to me, granted my only programming experience thus far is with PicBasic, Arduino, and some assembler on the quite obsolete 68HC11s, so very limited in the grand scheme of things. Never touched anything x86.So ultimately... it should be a matter of inserting the missing code into the overall binary file using a hex editor, write the resulting, hopefully complete and correct, binary to the flash chip, and in theory the card should be operable (at least in a limited capacity, assuming there's still a hardware fault on it of some sort which created the problem in the first place)?

author: amyk
author_rank: Super Contributor
timestamp: October 31, 2019, 02:13:23 am
content: Quote from: TheMG on October 31, 2019, 01:21:48 amReverse-engineering skills, not to be confused with programming... it usually starts with a few educated guesses and goes on from there.

author: fzabkar
author_rank: Super Contributor
timestamp: October 31, 2019, 03:30:50 am
content: Quote from: TheMG on October 31, 2019, 01:21:48 amYes, that would be my approach. I would suggest that you let us see your final compilation before committing it to flash.As for my methods, yours is not the first firmware which I have "reverse engineered", so it wasn't as difficult as it might otherwise have been. I'm not a programmer, but I can write simple programs. (I can also fry an egg, but that doesn't make me a chef.)That said, the only skills you need for this particular task are a familiarity with hexadecimal arithmetic and an ability to recognise patterns in the data. But that's still only the easy part. Disassembling the code is much more challenging -- that's where the real reverse engineering knowledge is. I confess that I haven't done much in that regard.As for tools, I used a freeware hex editor (HxD) for file viewing, carving, comparison and checksum calculation, and MSDOS DEBUG.EXE for disassembly.Edit:This is the repaired flash dump, AISI. I suggest you repair yours independently, just in case we don't agree.http://www.users.on.net/~fzabkar/temp/UPS_flash_contents_repaired.7z

author: TheMG
author_rank: Frequent Contributor
timestamp: November 05, 2019, 01:54:19 am
content: Well, I patched the flash bin file to the best of my abilities, then used the compare feature in HxD to compare with yours, and it claims the two to be identical.I guess the next thing to do now is to write that to the flash memory and solder it back onto the PCB and see what happens.The stuff in the BOOT module at offset 000F306F in the flash still intrigues me:Code:1) CPU SELF TEST2) SRAM MEMORY TEST3) NIC INTERNAL TEST4) FLASH ROM TEST5) LED BLINKING TEST6) READ DIP SWITCH STATUS7) NIC EXTERNAL LOOPBACK TEST9) REAL TIME CLOCK TESTC) REAL TIME CLOCK ADJUSTF) FLASH FUNCTIONALITY TESTA) I2C TESTThis would suggest some debugging functionality is built-in the BOOT module, but the million dollar question is... how can it be accessed/invoked?

author: amyk
author_rank: Super Contributor
timestamp: November 05, 2019, 02:41:56 am
content: Quote from: TheMG on November 05, 2019, 01:54:19 amLoad the file into a disassembler and do some tracing. You'll need to find how that code is reached.There is, of course, always the possibility that it's completely "dead code" and unreachable, left over remnants from an actual test ROM.

author: TheMG
author_rank: Frequent Contributor
timestamp: November 05, 2019, 02:52:38 am
content: Well, after knocking over my bottle of rosin flux all over the work bench and having a huge sticky mess to clean up...It's ALIVE!!!What's more, it let me configure the IP address without crashing.It is possible there's still a problem (faulty RAM maybe?) but this is a different firmware version from what was originally on the card so stuff relating to the IP address setting could simply be in a different, non-faulty, location in RAM.So it's too early to tell whether it's fixed for good or not.Next test is to chuck the card into one of my spare UPS and letting it run for a while to see if it really is stable or if it still has issues randomly crashing. For the time being, powering the card on the bench it seems good.

author: fzabkar
author_rank: Super Contributor
timestamp: November 05, 2019, 04:05:58 am
content: I just have one cautionary observation.The headers of the BOOT, FILE and MAIN modules each have a 4-byte parameter which is set to 0xFFFF in the payload, but which changes when the module is written to flash. This would suggest that this parameter is unique and is in some way dependent on the target PCB. Perhaps it is a security related CRC??In any case, I would be reluctant to write this flash dump to a different PCB, just in case it is indeed unique.As for a Debug Monitor, what is the function of jumper block JP1?

author: elie13
author_rank: Newbie
timestamp: January 11, 2024, 02:41:34 pm
content: Hellothanks for the information you provided, i have the same problem with the connectups-X card, please can you share the latest modification firmware?, did you try after to flash it normally through serial port with the manufacture firmware? thanks for your help