Keysight MSOX3104T Start-Up Problem! - Page 1
Forum section: electronics / testgear

author: Sal Ammoniac
author_rank: Super Contributor
timestamp: October 14, 2018, 09:49:23 pm
content: My backup oscilloscope, a Keysight MSOX3104T, doesn't complete its start-up sequence and appears dead.This is pretty much a brand new scope. I've had it for two years, but it's gotten very little use since it's my backup scope. It's been unplugged from the wall outlet for over a year. When I plugged it back in today and pushed the power button, it turns on the LED next to the  key and then turns on the LED next to the key, but it stops there and doesn't get any further. The screen itself is completely black.I can't try updating the FW because it doesn't get far enough to allow me to attempt that.Anyone seen this before? Is there a fix?

author: HighVoltage
author_rank: Super Contributor
timestamp: October 14, 2018, 09:59:12 pm
content: It is probably a NAND corruption and Keysight will fix it under warranty, even if the warranty has expired.

author: GlowingGhoul
author_rank: Regular Contributor
timestamp: October 14, 2018, 11:05:31 pm
content: EXECTLY the same thing occurred to me today, except my scope was last used 3 months ago. Please let me know how Keysight handles this for you.

author: Sal Ammoniac
author_rank: Super Contributor
timestamp: October 15, 2018, 02:06:32 am
content: Quote from: GlowingGhoul on October 14, 2018, 11:05:31 pmI plan on calling Keysight tomorrow. I'll let you know how it goes.I've found a number of other descriptions of this happening to people with MSOX3000 and MSOX4000 series scopes by doing a Google search. The common link between the incidents appears to be not using the scope for a period of several months. Why this should cause the scopes to not power up properly is beyond me. Perhaps there's some CMOS configuration memory that's kept alive by a rechargeable battery or supercap that discharges if the scope isn't used for a long period of time?

author: sibeen
author_rank: Frequent Contributor
timestamp: October 15, 2018, 02:13:49 am
content: Rushes over and turns on my 3024T.I probably haven't had it on for a month or so.

author: TheSteve
author_rank: Unknown
timestamp: October 15, 2018, 02:18:16 am
content: Quote from: Sal Ammoniac on October 15, 2018, 02:06:32 amSeems to just be a bit or two in the NAND flash memory that decides to alter itself. Older firmware doesn't recover from the issue very well, newer firmware does. Scopes have a 3 year warranty and Keysight is pretty good with warranty repairs.

author: Sal Ammoniac
author_rank: Super Contributor
timestamp: October 15, 2018, 04:00:11 am
content: Quote from: TheSteve on October 15, 2018, 02:18:16 amI wonder if there’s a way to force a firmware update if something like this happens?

author: TheSteve
author_rank: Unknown
timestamp: October 15, 2018, 04:21:44 am
content: Quote from: Sal Ammoniac on October 15, 2018, 04:00:11 amIt can most likely be recovered if you have a network card by booting an image via network and then updating firmware. This has been covered in the 2000/3000 hacking thread for the most part. I've never done it with a T series scope, only the 2000a/3000a.

author: Sal Ammoniac
author_rank: Super Contributor
timestamp: October 15, 2018, 01:41:58 pm
content: Quote from: TheSteve on October 15, 2018, 04:21:44 amUnfortunately, I don't have a network card. I was hoping for something simpler, like "Insert a USB drive with new FW in the USB slot and power-up the unit while holding down the X, Y, and Z keys".

author: Sal Ammoniac
author_rank: Super Contributor
timestamp: October 15, 2018, 04:53:45 pm
content: Quote from: GlowingGhoul on October 14, 2018, 11:05:31 pmI spoke to Keysight tech support this morning. They're very friendly and knowledgeable. He acknowledged that this is a known problem that affects these scopes and Keysight will fix it even if the scope is no longer under warranty.As HighVoltage and TheSteve said, it seems that this is a FLASH corruption issue that earlier versions of FW couldn't recover from.The tech said that if you can get it to complete the start-up sequence just once, you can load the latest FW and that will fix the problem. I plan to spend an hour or so tonight trying to get it to boot before I box it up and ship it back to Keysight for repair.As an embedded engineer, if it were me designing a piece of equipment like this I would have implemented a fail-safe way to update the FW even if the FLASH gets corrupted and the unit won't boot.

author: 2N3055
author_rank: Super Contributor
timestamp: October 15, 2018, 06:08:42 pm
content: Quote from: Sal Ammoniac on October 15, 2018, 04:53:45 pmThey did, in later version of firmware...

author: GlowingGhoul
author_rank: Regular Contributor
timestamp: October 15, 2018, 06:09:31 pm
content: I wasn't able to speak with a tech. They took my information and said I should hear from someone within 24hours. They had no idea what I was referring to regarding NAND corruption issues.My warranty ended 2 months ago, and I don't have a good feeling about this.

author: HighVoltage
author_rank: Super Contributor
timestamp: October 15, 2018, 08:00:32 pm
content: My 3000X scope was almost 2 years out of warranty and Keysight fixed it for free, after I had send them the scope.

author: Sal Ammoniac
author_rank: Super Contributor
timestamp: October 15, 2018, 08:11:46 pm
content: Quote from: 2N3055 on October 15, 2018, 06:08:42 pmI'm talking about a hardware fail-safe here. A FW fix obviously doesn't help if there's no way to update the FW!

author: 2N3055
author_rank: Super Contributor
timestamp: October 15, 2018, 08:58:37 pm
content: Quote from: Sal Ammoniac on October 15, 2018, 08:11:46 pmWell I guess they know how... I guess they setup secondary image or ECC or something. It can detect firmware corruption and starts backup boot procedure...

author: TheSteve
author_rank: Unknown
timestamp: October 15, 2018, 09:02:44 pm
content: The bootloader is in NOR flash, it loads the primary image from the NAND flash, if that fails it goes to a second image - in newer firmware anyway. So most of the time a network boot can recover the scope, unless some of the truly unique info was lost - model/serial, cal data, licenses etc. The newer firmware also handles remapping bad blocks so that both images are correct/clean.

author: GlowingGhoul
author_rank: Regular Contributor
timestamp: October 15, 2018, 09:37:02 pm
content: Quote from: TheSteve on October 15, 2018, 09:02:44 pmI'm a bit confused. Are you saying that you need to network boot vs normal boot to recover, or that recovery happens automatically, in which case what is the relevance of network boot? I have ethernet on this scope.

author: TheSteve
author_rank: Unknown
timestamp: October 15, 2018, 09:55:00 pm
content: Quote from: GlowingGhoul on October 15, 2018, 09:37:02 pmYou'd need to open the scope and connect to the serial debug header. From there you can interact with the bootloader and tell it to tftp a prepared image via a network connection.

author: Sal Ammoniac
author_rank: Super Contributor
timestamp: October 16, 2018, 04:57:37 pm
content: Spent 45 minutes last night repeatedly pushing the power button. No joy. I'll try the same tonight and if still no joy will pack it up and ship back to Keysight tomorrow morning. They quoted a 12-14 day turn-around time on the repair.

author: GlowingGhoul
author_rank: Regular Contributor
timestamp: October 16, 2018, 06:26:43 pm
content: It's been over 24 hours since I called Keysight and was told I'd be called back within that time. Not good.

author: GlowingGhoul
author_rank: Regular Contributor
timestamp: October 16, 2018, 08:55:09 pm
content: Just a quick update on Keysight's customer service. I just called, since they never called me back.Bottom line, they will not assist, period. I'm on my own, 60 days out of warranty.They denied knowing anything about 'corrupt nand', said there is no service note, and there is no 'known defect' of that type with any model.When I asked about paid repair options, they demanded to see my origianl invoice, depsite purchasing the scope from Newark, and registering it with them the day it arrived, and purchasing numerous options that Keysight had to process.Never again. A $5000 paperweight they refuse to repair. For what it's worth, don't tell them you are an individual, you MUST be a company if you want decent customer service. The moment I told them I had no company name, this was my personal scope, their attitude immediately changed from eager to help to suspicion. What the hell happened to Keysight?

author: JonM
author_rank: Unknown
timestamp: October 16, 2018, 09:13:52 pm
content: Quote from: GlowingGhoul on October 16, 2018, 08:55:09 pmI am fairly certain that Keysight will fix this, but it might not get moving until Daniel Bogdanoff sees this thread. There have been multiple posts about the NAND corruption and it seems that Keysight fixed them all, even if past warranty period.I just powered up my personally owned DSOX3024T to be sure that it is OK after sitting for a while, but I have recent firmware so it should not be an issue. I do wonder if I should get a service contract when my warranty runs out in another year. However, it's probably expensive for a lightly used, personally owned, non-profit instrument.

author: GlowingGhoul
author_rank: Regular Contributor
timestamp: October 16, 2018, 09:26:30 pm
content: I called back, livid. Keysight agreed to evaluate the unit for repair for a fee of $311, but I won't have to pay that until they give me the full quote for the repair (how generous), somewhere around $2000.I was told there is no service note, no known widespread problem, no NAND fault repairs out of warranty.

author: sibeen
author_rank: Frequent Contributor
timestamp: October 16, 2018, 09:33:50 pm
content: GlowingGhoul, perhaps you should mention where you are based. Someone, probably Daniel, from Keysight will eventually see this or be pointed to it. They'd need to know where you are from so the arse kicking can commence.

author: GlowingGhoul
author_rank: Regular Contributor
timestamp: October 16, 2018, 09:35:19 pm
content: Quote from: sibeen on October 16, 2018, 09:33:50 pmI'm in the US, and was dealing with customer 'service' at Keysight HQ.

author: TheSteve
author_rank: Unknown
timestamp: October 16, 2018, 09:43:42 pm
content: Assuming it is a NAND corruption issue(likely) then hopefully Keysight will fix it for free. If they won't do it free then I am 99% sure I can help you recover it.

author: GlowingGhoul
author_rank: Regular Contributor
timestamp: October 16, 2018, 09:47:41 pm
content: Quote from: TheSteve on October 16, 2018, 09:43:42 pmI appreciate that. If they come back with a large repair quote, as I expect, I'll still have to pay $311 to get the scope back. Hopefully I can get it running again so I can sell it and move on to another brand.

author: bicycleguy
author_rank: Frequent Contributor
timestamp: October 16, 2018, 10:11:07 pm
content: I'll take that nasty MSOX3104T off your hands.How much?

author: tautech
author_rank: Super Contributor
timestamp: October 16, 2018, 10:51:21 pm
content: Quote from: GlowingGhoul on October 16, 2018, 09:47:41 pmYou just might want to follow this:https://www.eevblog.com/forum/testgear/test-equipment-anonymous-(tea)-group-therapy-thread/msg1830599/#msg1830599https://www.eevblog.com/forum/testgear/test-equipment-anonymous-(tea)-group-therapy-thread/msg1887386/#msg1887386https://www.eevblog.com/forum/testgear/test-equipment-anonymous-(tea)-group-therapy-thread/msg1894256/#msg1894256https://www.eevblog.com/forum/testgear/test-equipment-anonymous-(tea)-group-therapy-thread/msg1897616/#msg1897616Yes you read it right, member URI bought a busted Agilent DSO-X 3034A and is gunna get it repaired free by KS !You need contact Daniel to get this sorted if the call center staff have no idea of what's going on:https://www.eevblog.com/forum/profile/?u=114509

author: GlowingGhoul
author_rank: Regular Contributor
timestamp: October 17, 2018, 05:08:50 pm
content: I see they did. However, pointing that out got me nowhere.$311 for a diagnosis, then $XXXX for the repair.I guess my mistake was buying a brand new one from an authorized reseller for $5k, rather than buying a broken boat anchor for $100 from eBay.

author: kcbrown
author_rank: Frequent Contributor
timestamp: October 17, 2018, 05:25:17 pm
content: Quote from: 2N3055 on October 15, 2018, 06:08:42 pmOf course, that doesn't help you if the fail-safe part of the NAND also gets corrupted ...I've always thought that the best way to do these things is to have a known good version (i.e., the version that was available at the time of manufacture) of the firmware in ROM that one can boot from in the event the NAND image gets toasted somehow. Not only would that provide an alternate boot in the event that the firmware upgrade fails, it provides a hardware-level failsafe in the event the NAND chip itself fails.

author: nctnico
author_rank: Super Contributor
timestamp: October 17, 2018, 05:28:29 pm
content: Quote from: kcbrown on October 17, 2018, 05:25:17 pmIt is more likely the original error correction isn't good enough to catch a few bit flips.

author: GlowingGhoul
author_rank: Regular Contributor
timestamp: October 17, 2018, 05:52:06 pm
content: I've spoken to 3 different people at Keysight regarding this, they all consistently claim:1) There is no service note regarding NAND corruption or repairs out of warranty2) There is no 'known issue' regarding NAND/firmware corruption3) There is no coverage for any known issue beyond the warranty period. Either pay the $311 diagnosis fee+repair parts and labor cost, or it will not be repaired.

author: Fgrir
author_rank: Regular Contributor
timestamp: October 17, 2018, 06:15:04 pm
content: This threadhas a lot of the original discussions, and interestingly Daniel at the time pointed out that the out-of-warranty repair was not a forever thing:Quote from: Keysight DanielBogdanoff on May 23, 2016, 04:56:45 pmFunny thing is as I was re-reading it I saw GlowingGhoul was well aware of the issue and not so full of sympathy:Quote from: GlowingGhoul on May 24, 2016, 06:06:14 pmKarma is a bitch...

author: GlowingGhoul
author_rank: Regular Contributor
timestamp: October 17, 2018, 06:24:52 pm
content: Quote from: Fgrir on October 17, 2018, 06:15:04 pmTell me, before I demonstrate what an intellectually dishonest fool you are, what was the fault I was referring to? Was it a known and widespread design defect that triggered by the innocuous behavior of leaving the scope off for a time? Well? Go ahead and tell everyone what sort of out-of-warranty fault I was referring to.

author: GlowingGhoul
author_rank: Regular Contributor
timestamp: October 17, 2018, 06:38:54 pm
content: I'll fill in the details. Keysight replaced a defective unit (for the corrupt nand issue) with a new one at no charge. User said he didn't try the replacement unit for6 monthsafter receiving it. When he did try it, it had a PLL error, which Keysight would not fix for free.THAT is what my comment was in response to, in context.

author: Fgrir
author_rank: Regular Contributor
timestamp: October 17, 2018, 06:40:50 pm
content: Admittedly his situation was more complex than yours, but I'll stand by the comparison to someone with an out-of warranty situation who didn't choose to pay the "insurance" of the extended service contract.I'm not trying to tear you down, and I'm sorry that you took it that way. I really do hope you get your scope fixed for free.

author: GlowingGhoul
author_rank: Regular Contributor
timestamp: October 17, 2018, 06:49:48 pm
content: Quote from: Fgrir on October 17, 2018, 06:40:50 pmI would not be on here complaining about them not fixing an out of warranty failure that didn't involve a multi-year, thoroughly known and documented design flaw. The proper thing to do here is to acknowledge the problem and offer free extended coverage on that fault alone, instead of trying to cover it up by providing limited free repairs while giving themselves the ability to absolve themselves of responsibility when it suits them.

author: kcbrown
author_rank: Frequent Contributor
timestamp: October 17, 2018, 07:01:54 pm
content: Quote from: nctnico on October 17, 2018, 05:28:29 pmNo doubt. But with sufficiently bad luck, the area which houses the previous version of the firmware may also be corrupted by the same mechanism.I suppose with sufficient ECC bits per data word, you'll drop the probability of an uncorrectable error low enough that it no longer matters. But either your ECC mechanism is stored in the flash itself, in which case you've got a bootstrapping problem (how do you deal with the situation where the ECC code itself gets corrupted?), or it's in hardware/ROM. It sounds like, in Keysight's case, it might live in the flash itself (though it may be that the measures they've taken involve periodic scans of the flash to look for errors and to actively rewrite the afflicted pages). So you really do want a hardened alternate boot mechanism that will at least make it possible to restore the flash contents to sanity. And if you're going to go to that trouble, you may as well make that mechanism be a ROM with a known-good image of your firmware, so that you're guaranteed to at least be able to use the scope even if the flash is so compromised that it is no longer writable.

author: rsjsouza
author_rank: Super Contributor
timestamp: October 17, 2018, 07:09:50 pm
content: This comparison is still valid, and quite ironic considering the vitriolic tone of your words then.Quote from: Fgrir on October 17, 2018, 06:40:50 pmQuote from: GlowingGhoul on May 26, 2016, 08:34:05 pm

author: GlowingGhoul
author_rank: Regular Contributor
timestamp: October 17, 2018, 07:19:22 pm
content: My 'tone' makes the comparison valid? You see no difference between a company acting responsibly to address a serious design flaw, and a company declining to repair a non-design flaw related failure out of warranty?Apparently my 'tone' has driven you to disregard logic out of a desire to criticize me because I offended your delicate sensibilities.

author: Pinkus
author_rank: Frequent Contributor
timestamp: October 17, 2018, 09:58:52 pm
content: QuoteAs my DSOX4000 scope starts to often need multiple on/off switches to start up (it often hangs during boot with only the Ref LED illuminated) I am afraid of an upcoming NAND problem.(btw: I would appreciate any ideas how I might minimize any future/potential NAND problem now; would a backup help?)As Daniel explicitly says "prior 2016": two questions are coming up for me:1) What was changed after 2015? New NANDs (or different manufacturer)?2) I assume the NAND is a separate chip. Would it be possible to desolder and change the NAND against a new one?

author: GlowingGhoul
author_rank: Regular Contributor
timestamp: October 17, 2018, 11:24:21 pm
content: Upgrade to the latest firmware. They've been incorporating measures to minimize the chances of corruption preventing a boot.

author: Sal Ammoniac
author_rank: Super Contributor
timestamp: October 18, 2018, 03:56:49 am
content: Just spent another hour attempting to get it to boot without luck. I'll be sending it back tomorrow.Perhaps I should ask them to replace the on/off switch too--it's gotten quite a workout in the past few days.

author: Pinkus
author_rank: Frequent Contributor
timestamp: October 18, 2018, 06:24:17 am
content: Quote from: GlowingGhoul on October 17, 2018, 11:24:21 pmI've already been using the latest firmware for a year. Since DSOX3000T are also affected, the newer firmware obviously is not a miracle cure. Possibly a time bomb is sleeping in all DSOX scopes. Hence my question about the possibility to exchange the NAND chip.

author: Keysight DanielBogdanoff
author_rank: Unknown
timestamp: October 19, 2018, 10:50:20 pm
content: PM sent. I'll try to get you fixed up.Our service centers are excellent 99.9% of the time, but every once in a while something slips through. I'm happy when I can catch it and help here.We've done extensive work on making the NAND more robust, I highly recommend that everyone update their firmware. This includes a better backup scheme and a recovery image when there are issues detected. Old firmware won't have this, new firmware does.I'll also say that if your scope is out of warranty, the service centers will almost always quote a charge to inspect/repair. If it turns out to be a NAND issue, you won't get charged. But, they can't quote free because they don't know for sure that it is NAND. At least, that's how it was explained to me.I'll get info from you through PMs and try to work some magic on the back end.-Daniel

author: Sal Ammoniac
author_rank: Super Contributor
timestamp: October 31, 2018, 08:55:26 pm
content: I got my scope back from Keysight yesterday and it now boots normally.The repair turn-around wasn't too bad. I shipped the scope on October 22nd and got it back on October 30th, so a little over a week door-to-door.The service order details said they replaced the acquisition system board. This was more than I expected was needed here. I was expecting them to update the flash to the latest firmware revision using an internal JTAG or debug header.This whole episode leaves a bad taste in my mouth. I'm always going to wonder if the scope is going to boot up every time I need to use it. Based on this experience, my next scope probably won't be a Keysight.

author: Keysight DanielBogdanoff
author_rank: Unknown
timestamp: October 31, 2018, 09:20:20 pm
content: Hi Sal,Our standard process is to swap the board out, that cuts down on theveryexpensive repair time. I've talked through it more on a couple other threads here, but troubleshooting and repair time can easily exceed the price of an InfiniiVision scope, so we just do a swap and refurb the board at the factory. As with all electronics, we can't guarantee that it'll be trouble free forever. But, the new NAND protection in our firmware is orders of magnitude more reliable than previous versions (internal data backs up that claim).Again, I'm sorry for the trouble you had and the time you spent troubleshooting. If you have any issues, feel free to PM me and we'll get you taken care of!

author: sibeen
author_rank: Frequent Contributor
timestamp: October 31, 2018, 09:46:02 pm
content: Every time this, or a similar thread, comes up in the list of active threads I turn around and turn on my 3024T

author: Sal Ammoniac
author_rank: Super Contributor
timestamp: October 31, 2018, 10:05:30 pm
content: Quote from: Keysight DanielBogdanoff on October 31, 2018, 09:20:20 pmI wonder how the original design made it through the design review process? This is really a rhetorical question as I wouldn't expect you to know.I bought this scope directly from Keysight, so I'm also wondering why they didn't notify me via email (they had my address from the order) to update the FW? I've gotten written notices from car companies every time they have a recall (and some of these were merely annoying issues, not just safety-related issues)--perhaps Keysight should do something similar.

author: JonM
author_rank: Unknown
timestamp: November 03, 2018, 12:10:00 am
content: Quote from: Sal Ammoniac on October 31, 2018, 10:05:30 pmI bought my scope through TestEquity but I have it registered under my MyKeysight account which does cause me to get some emails about the products I own. I believe that I have received notifications of new software for the scope but don't remember any that suggested it was an urgent update (I learned of the NAND issue on this forum). Checking one of the messages I see that I asked for certain updates:Your current email subscription Frequency SelectionForum update Weekly My assets onlyProduct Manual Weekly My assets onlyApplication notes Weekly My assets onlySoftware update alert Weekly My assets onlyNew FAQs Weekly My assets onlyI will note that I see a lot of forum updates for "Truevolt Series 6.5 & 7.5 Digit Multimeters" that have nothing to do with a DMM.