Agilent MSOX4054A repair (NAND corruption issue) - Page 1
Forum section: electronics / testgear

author: hotacid
author_rank: Newbie
timestamp: January 13, 2020, 08:22:16 am
content: Hi folks! Longtime lurker here coming out of the darkI recently acquired a used MSOX4054A scope from an auction only to find it doesn't boot when I try to turn it on. The unit is in great shape so I don't suspect anything physical has damaged it.This is what happens:1. I push the power button to the "on" position. I hear the fan turn on.2. The front panel LEDs all turn on and off again once very briefly3. About 1s later, the "Ref" button LED turns on and stays solid4. About 1s later, the "Math" button LED turns on and stays solid5. About 3s later, the "Digital" button LED turns on and stays solid6. Nothing more happens past this. The scope stays stuck foreverAfter hours of searching the web on this issue, I came to understand there is a known issue with the 2000X/3000X/3000T/4000X series scopes where the main NAND becomes corrupted over time and this prevents the unit from booting. The descriptions of other peoples issues sounds very similar to my units behavior so I suspect I have the same problem.I found some mentions of Keysight fixing this problem for free so of course I tried that path first... a week later and several emails + phone calls later, Keysight told me that it won't be covered for me and wouldn't give a reason why. Strangely, the applications engineer I spoke to was also surprised that it wasn't covered but even a 3-way call with him and the repair department didn't help. They wanted $4500 USD so I politely declined.So now I'm trying to fix this unit at home. Fortunately, there is a quite helpful thread about hacking these series of scopes which contains very valuable information on how to potentially repair this thing. Yes, I read all 97 pages.https://www.eevblog.com/forum/testgear/dsox2000-and-3000-series-licence-have-anyone-tried-to-hack-that-scope/I've gotten the unit apart and I have access to the serial connection from the SPEAR600-2 CPU. A serial log of the unit booting seems to all but confirm some memory corruption has occurred. Here's a snippet of the serial output (full log attached).QuoteI've been following along the footsteps of one user (titiris - thank you!) who reported he was able to fix his NAND corruption by transferring the unpacked firmware kernel image 'nk.nb0' to the scope over the serial connection (using YMODEM mode), booting the image, which then boots from an external USB drive. Once the Infiniium software is up and running, a normal firmware update procedure can be carried out through the scope GUI to repair the damaged files. His explanation is here:https://www.eevblog.com/forum/testgear/dsox2000-and-3000-series-licence-have-anyone-tried-to-hack-that-scope/msg2136181/#msg2136181However, I'm hitting an issue with this method. It seems that the YMODEM image transfer always stops abruptly when it gets to ~75% complete. No error messages or warnings are printed. I also tried following along Frankbuss's method by using TFTP but this hits the same issue... the transfer is working fine until 61466628 Bytes are transferred (out of 81508824 Bytes total) and then it just stops. It looks like the unit is hanging. Is the image too big and the scope ran out of memory? I'm not sure what is going wrong.https://www.eevblog.com/forum/testgear/dsox2000-and-3000-series-licence-have-anyone-tried-to-hack-that-scope/msg260917/#msg260917Has anyone tried this on a 4000X series scope? Any suggestions would be greatly appreciated!

author: TheSteve
author_rank: Unknown
timestamp: January 13, 2020, 06:24:39 pm
content: I've never had the chance to try repairing a 4000x series, only the 2000/3000. Your file does seem large though compared to the ones for the 2000/3000.I dream of finding a corrupted 4000x(or even 6000x) series some day for a great price.

author: hotacid
author_rank: Newbie
timestamp: January 14, 2020, 04:58:02 am
content: Yeah the image size is about double that of the 2000/3000 series. But the instrument must have enough RAM to hold the transferred image, right? I'll see if I can split the image in two and do two transfers.Another problem I have is that I can't find the old firmwares for this scope. It was mentioned in titiris' guide that the firmware version I transfer over to the unit has to be similar to the (corrupted) one on the NAND, or the boot may not get far. If anyone has one sitting on their machine, I'd love to get my hands on it! Here's a complete list of the firmwares. I have only managed to find 7.30 and 4.08 from searching the internet.File NameRelease DateInstrument Software Version4000XSeries.7.30.2019051435.ksx29 May 20197.30.20190514354000XSeries.7.20.2017102615.ksx26 Oct 20177.20.201710264000XSeries.7.11.2017061227.ksx12 Jun 20177.11.201706124000XSeries.7.10.2017042903.ksx29 Apr 20177.10.201704294000XSeries.07.10.2017041132.ksx11 Apr 20177.104000XSeries.04.08.2016071801.agx27 Jul 20164.084000XSeries.04.07.2016040802.agx20 Apr 20164.074000XSeries04.05.2015051200.agx27 May 20154.064000XSeries04.05.2015021000.agx16 Feb 20154.054000XSeries.04.00.2014101303.agx01 Nov 20144.004000XSeries03.22.2014052101.agx21 May 201403.22.20140521014000XSeries.03.21.20140110001.agx21 Jan 201403.21.201401100014000XSeries.03.20.2013082300.agx03 Sep 201303.20.20130823004000XSeries.03.12.2013041700.agx23 Apr 201303.12.20130417004000XSeries.03.11.2013030100.agx01 Mar 201303.11.20130301004000XSeries.03.10.2013020700.agx12 Feb 201303.10.20130207004000XSeries.03.01.20121212001.agx12 Dec 201203.01.201212120014000XSeries.03.00.201209XXXX.agx?? Sep 20123.00

author: TheSteve
author_rank: Unknown
timestamp: January 14, 2020, 07:15:24 am
content: With a 2000/3000 the firmware you transfer over does need to be the same version on the scope or very close or it won't boot all the way(or at all). Is the scope Agilent or Keysight branded, you can use that to give you a hint of a version to try. I have no 4000x series firmware myself.I still am wondering about the image you're using, I don't think it should be double the size.

author: hotacid
author_rank: Newbie
timestamp: January 14, 2020, 08:24:48 am
content: Quote from: TheSteve on January 14, 2020, 07:15:24 amYes, I'm quite worried that will be the case when I do figure out how to transfer it.The scope is Agilent branded. Looking at its service history, Keysight performed a repair (the scope wouldn't fully power on!) by replacing the acquisition board and calibration on Feb 11 2015. There is also an interesting note that says the owner declined to have Service Note MSOX4054A-02 completed around the time it was being repaired. This service note is recommending a firmware upgrade if the version on the instrument is 3.11 or older. The fact that the owner declined a firmware update in 2015 makes me think it was probably not updated past this point. So the scope may have an ancient 3.11 or older version on it... not good.I think 80MB is correct because the "Declassification of the InfiniiVision X-Series Oscilloscopes" document you can find online shows the Windows CE portion being this large.Splitting the image in two didn't work. The first transfer was fine (41943040 bytes) but the second transfer hit a hang after byte 19522932 had been transferred. I also tried placing it in at a different starting address in memory (0x0 instead of 0x361000) but this likewise hit a hang after 65010380 bytes.

author: tv84
author_rank: Super Contributor
timestamp: January 14, 2020, 10:21:56 am
content: Quote from: hotacid on January 14, 2020, 04:58:02 amThis is a complex recovery. You need to have more assurances in order to succeed. The equipment has 128 MBytes RAM as you can see in your boot log.Aren't you overwriting the prog that is running, when loading into RAM?Is RAM all OK?First step, I suggest you do a methodic mapping of your RAM space in order to understand if your loading space is valid. Take a 10 MB chunk and load it at offset 0, 10 MB, 20 MB, 30 MB, 40 MB, 50 MB, etc... and see where it hangs.Can you stop the bootloader? That would be great. That could allow us to dump NAND.Can you dump the bootloader (NOR Flash 512kB)?

author: TheSteve
author_rank: Unknown
timestamp: January 14, 2020, 05:45:28 pm
content: You then likely do need firmware 3.x to have a chance at recovering the scope. Unless you find a way to recover it we haven't thought of yet.

author: PA0PBZ
author_rank: Super Contributor
timestamp: January 14, 2020, 06:53:51 pm
content: Your bootlog suggests that it is trying to load 0x4D6BA80 bytes into RAM, which equals to 81181312 decimal bytes. This is smaller than your image and my best guess is that it is indeed a previous version since they never shrinkIf you can locate older versions I'd take the one with an image size of 81181312.

author: Pinkus
author_rank: Frequent Contributor
timestamp: January 15, 2020, 10:34:38 am
content: I have all firmware version starting with Keysight branding, thus 4.00 ++I checked the size of nk.bin of all firmware versions, unfortunately no match so you do not have to bother with 4.xx versions:4.00: 80922891 byte4.01-4.04: no such, regarding the release notes4.05: 81539207 byte4.06: 81413855 byte4.07: 79718035 byte4.08: 81031923 byte7.10 and newer are all way largerUnfortunately I do not have any version below 4.00 available. There once was a helpful HP/Agilent group on Yahoo where you could ask for such help. You might try it there if the group still exists. Possibly it moved away from Yahoo to somewhere else ... i have some vague memory about this.

author: Keysight DanielBogdanoff
author_rank: Unknown
timestamp: January 15, 2020, 10:07:26 pm
content: I'm going to check in as well and see what the internal folks have to say.

author: hotacid
author_rank: Newbie
timestamp: January 17, 2020, 05:54:41 am
content: Quote from: tv84 on January 14, 2020, 10:21:56 amtv84:You make an excellent point. It would make sense that I see the scope hang if I'm overwriting the running program. I will try mapping out the memory as you suggested.Yes, I can stop the bootloader. There are actually two stages where I can stop it: the first is at the U-Boot monitor prompt, where I am able to execute a large number of possible commands. Please see the attached log for the list. The second is after the "PBOOT" primary boot loader image, which appears to be a Windows CE 6.0 standard bootloader. From there, I have four choices (again, see log). One very interesting one is an option to boot over ethernet. I tried this but the scope is waiting for some response from a host machine over the network that appears to be related to a "Windows CE 6.0 Platform Builder" tool that I need to see if I can get a copy of.The U-Boot monitor seems to give full control over the CPU resources. I just have to hack around and experiment to figure out how to use it all. It should be possible to dump arbitrary memory regions and run arbitrary code. The only issue is that I'm mostly a hardware guy so it will take me some timeQuote from: Pinkus on January 15, 2020, 10:34:38 amPinkus:Thank you very much for checking on that for me. That is strong evidence that the scope is indeed on a 3.xx build.Quote from: Keysight DanielBogdanoff on January 15, 2020, 10:07:26 pmDaniel:I would be thrilled if you could help out! An old firmware image, maybe some hint on possible recovery mechanisms, anything...

author: PA0PBZ
author_rank: Super Contributor
timestamp: January 17, 2020, 08:00:21 am
content: Quote from: hotacid on January 17, 2020, 05:54:41 amI compiled a tool that you can use on a windows machine many moons ago:https://www.eevblog.com/forum/testgear/dsox2000-and-3000-series-licence-have-anyone-tried-to-hack-that-scope/msg1022267/#msg1022267The story behind it:https://www.eevblog.com/forum/testgear/dsox2000-and-3000-series-licence-have-anyone-tried-to-hack-that-scope/msg1022248/#msg1022248

author: TheSteve
author_rank: Unknown
timestamp: January 17, 2020, 08:48:33 am
content: CELoader is a tool I nearly always use during a recovery,

author: tv84
author_rank: Super Contributor
timestamp: January 17, 2020, 09:40:25 am
content: Quote from: hotacid on January 17, 2020, 05:54:41 amThose are good news. Go into NAND susbsytem and see what commands are available.Attach them here.

author: Keysight DanielBogdanoff
author_rank: Unknown
timestamp: January 17, 2020, 05:12:27 pm
content: I will be following up via PM, it sounds like there's a good chance it's the NAND.

author: hotacid
author_rank: Newbie
timestamp: January 21, 2020, 02:45:56 am
content: All: Daniel has reached out to me and it's looking like the scope probably does have the known NAND corruption issue, and if so, it will be repaired by Keysight at no cost. I can't turn up an offer like that so I'm ending my repair attempt of it.Quote from: tv84 on January 17, 2020, 09:40:25 amtv84:Please see the attached log. I dumped the NAND commands and also tried reading a few pages but it always returns all 0xFF for every byte. I've tried reading pages 0 to 3.Quote from: PA0PBZ on January 17, 2020, 08:00:21 amPA0PBZ:Thanks for the tool! I tried it out and it worked to send the image I had for v4.08 of the firmware. The scope got further than I've ever seen it get - it started booting it and the front panel LEDs went through more steps than before - but it must have hit a problem during the boot because eventually the LED sequence went into an infinite cyclic pattern. Unfortunately, there's no more serial output after it starts trying to boot it so I can't say what went wrong. Most likely it's because the firmware version I'm trying to boot is quite different from the 3.xx version already on it. I attached my boot log.

author: Pinkus
author_rank: Frequent Contributor
timestamp: January 21, 2020, 09:47:05 am
content: Quote from: hotacid on January 21, 2020, 02:45:56 amCongratulation. It looks like everything's gonna turn out just fine for you.However, I'm surprised the Keysight Repair Center turned down the free repair without even seeing the unit. At least they should have offered to check it. Did Daniel have anything to say about this?

author: hotacid
author_rank: Newbie
timestamp: January 22, 2020, 06:40:05 am
content: Quote from: Pinkus on January 21, 2020, 09:47:05 amNot specifically, no. Maybe it was a misunderstanding on one or both ends of the call. These things happen

author: Zucca
author_rank: Unknown
timestamp: January 22, 2020, 07:06:49 am
content: In my experience Keysight is a company you can trust, and they trust you always unless there are evidences you are a stupid crook.I hope all the company could go close to the Keysight standard.

author: HighVoltage
author_rank: Super Contributor
timestamp: January 22, 2020, 09:25:27 am
content: Quote from: hotacid on January 21, 2020, 02:45:56 amI had two scopes with NAND problems repaired by Keysight over the years and never had an issue.Keysight is an amazing company, that has always taken care of any problems I had.Great that it worked out for you as well.And great that we have Daniel here.

author: jake111
author_rank: Regular Contributor
timestamp: February 08, 2020, 08:23:22 am
content: For what it's worth, I had DSO-X3054A with the same problem. Mine has recurring NAND corruption issue even after I fixed it, I have corrupt NAND blocks being reloaded on almost every startup (visible in the serial console). Originally I was on 2.35 firmware which bricks when corrupt, it would become corrupt after being off for only a few minutes.It was stuck at the infinitely looping lights on the front panel even after using bootp command to load nk.bin from my computer and then using the NAND WRITE to write it to the NAND (this could also be accomplished by using ymodem transfer thru serial but MUCH slower). I suddenly realized that this message about failing to program the FPGA was important... So after programming NAND (which I had to do every few boots because it would become corrupt again) I then transferred FPGA3000A.bin (for 3000 series) into memory and programmed the FPGA with it, then without rebooting, booted with the boot command. Viola! My machine then proceeded to boot from my 2.35 USB boot disk, and I then hastily upgraded it to 2.50.Now it boots reliably but on almost every boot it notes corrupt NAND blocks and rewrites them. I think my NAND is not long for this world, but thought I would mention this trick in case someone else gets stuck at the infinitely looping lights and can't figure out why. It seems that the FPGA has to be programmed in order for the infiniivision launcher to proceed far enough to get the machine to boot from USB. I assume that the FPGA data must normally be stored in NAND and this is why mine lost it. NAND alzheimers is a terrible thing.

author: EE-digger
author_rank: Frequent Contributor
timestamp: February 08, 2020, 09:24:35 pm
content: Thanks for that valuable lesson !I wonder if we should have a sticky area which mirrors/archives all things related to the "resurrection" of DSOX, MSOX models.

author: jake111
author_rank: Regular Contributor
timestamp: February 08, 2020, 11:35:44 pm
content: From what I have learned I think it is possible to resurrect even if the flash is inadvertently corrupted or erased. You just need to know appropriate memory locations and then use hardware level interface to get it in there. It appears that you need only the boot loader (p500?) to regain console and network access if this is broken, and from there you can load the appropriate OS binary and boot far enough to revive it from USB. I found a lot of conflicting information in the 2000/3000 series thread about how to actually construct a USB boot disk, there seem to be many methods and formats for the .lnk file but I believe only one of them is correct and others simply work because of intact OS images on the machine. I know there are variances between firmware versions but based on my experience you should only be using the latest or later firmwares which can recover from these known NAND issues.Fortunately it sounds like Keysight has acknowledged the NAND flaw and are repairing it but I assume that sooner or later this will not be the case as all products fall out of support at a certain point. At this point the information on how to recover these scopes will be very important!

author: TheSteve
author_rank: Unknown
timestamp: February 09, 2020, 04:57:30 am
content: If the flash is fully erased you will lose factory data - cal data, serial #, licenses etc.

author: jake111
author_rank: Regular Contributor
timestamp: February 13, 2020, 03:17:39 am
content: Quote from: TheSteve on February 09, 2020, 04:57:30 amI'm confused on mine because it suffered the NAND issue yet the serial, model number and "license model number" are gone. Do you know if anyone has successfully rewritten these? I found the DLL with SCPI commands in nk.bin but its obfuscated and it looks like I'm going to have to walk the code to try and get strings out of it... Argh

author: TheSteve
author_rank: Unknown
timestamp: February 13, 2020, 05:03:16 am
content: If the bad block happened to be the one with the secure data stored on it then those items will be gone. I've had one that lost all license data.The model and serial # are stored in some encrypted form I do believe. I know on many Keysight products the model/serial can be set via a scpi command but I believe they usually need a factory flag to be set which allows it to be programmed one time.

author: jake111
author_rank: Regular Contributor
timestamp: February 13, 2020, 05:36:08 am
content: Well I don't think Keysight will be willing to help with this so I am going to have to walk through the code and see if the information I need is there. I suspect it is based on what I've found so far1) RegisterNativeSecureStorageCallbacks2) SecureStorage_Add3) SecureStorage_Clear4) SecureStorage_Free5) SecureStorage_ModelNumber6) SecureStorage_Read7) SecureStorage_ReadCommonAreaSecureStorage_Retrieve9) SecureStorage_SerialNumber10) SecureStorage_WriteIt appears that there is the ability to write to the secure storage in these DLL's. Will take some time to figure out. I'll post back when I do. Whether or not it can be done with SCPI command is still a mystery as those DLL's appear to be obsfucated so that will be a piece of work since this is not at all my specialty.

author: TheSteve
author_rank: Unknown
timestamp: February 13, 2020, 05:53:09 am
content: I know with other instruments I have seen they do not have the ability to encrypt the model/serial, only decrypt it. There is a tool which encrypts the data and sends it via SCPI.The command format is often something like "SecureDataTool_<Client>.exe Initialize-ModelNumber" and "SecureDataTool_<Client>.exe Initialize-SerialNumber <SerialNumber>".When the 2000/3000a series first came out there was a "leak" with the firmware update which included the tool to generate a model/serial #. You had to extract the binary and run it on the scope itself, but it was there. You could use the keys the binary contained to generate your own signed license files. After the leak Keysight changed the private key so it only worked with really old scopes. Being as you have have no keys at all it is certainly worth having a look at. The firmware update was version 1.10 - released before the 4000 series even existed I am sure.

author: jake111
author_rank: Regular Contributor
timestamp: February 13, 2020, 06:43:31 am
content: Quote from: TheSteve on February 13, 2020, 05:53:09 amVery good point - Thanks Steve!!!

author: wp_wp
author_rank: Regular Contributor
timestamp: September 27, 2020, 02:58:25 pm
content: Quote from: jake111 on February 08, 2020, 08:23:22 amhi,jake111how did you program the FPGA in P500?

author: analogRF
author_rank: Super Contributor
timestamp: February 05, 2021, 04:55:00 am
content: Quote from: wp_wp on September 27, 2020, 02:58:25 pmsame question here

author: kimwu
author_rank: Newbie
timestamp: March 14, 2021, 08:55:39 am
content: hi everyoneif you have same issue like thisyou can try below method as linkhttps://www.eevblog.com/forum/testgear/dsox2000-and-3000-series-licence-have-anyone-tried-to-hack-that-scope/msg2953340/#msg2953340

author: Jedi
author_rank: Newbie
timestamp: January 05, 2024, 08:10:02 am
content: Hello, I am a user of DSO 4104A.I also encountered the Boot NAND error, and when I tried to download the bootloader, it stopped at around 65-70% during the upload.Have you resolved this issue?I have the same issue as you. I have obtained firmware version 4.08

author: Stingy49
author_rank: Newbie
timestamp: February 26, 2024, 05:14:22 pm
content: Quote from: Jedi on January 05, 2024, 08:10:02 amJedi, I used the recover.bin method from the post above you using 4.08 on a 4104A. Boots into a screen which allows you to touch/click/hit enter and select a ksx/agx to be loaded on to the scope. If you were trying to use the "boot from flash drive" method, I would wipe the drive and make sure it only has the ksx/agx file on it. This method requires being hooked into the internal serial port if that isn't clear.-Derek