Tektronix MDO3014 failure, MDO3000 series - Page 1
Forum section: electronics / testgear

author: GridWork
author_rank: Contributor
timestamp: February 04, 2021, 01:47:32 pm
content: So 5 years ago I took some hard earned money an plunked it down on a brand spanking new Tektronix MDO3014 scope plus the logic analyzer expansion. Great running scope, did everything I needed it to until this last week. Fast forward to current time, the scope would just keep resetting. Turn it on, it would run for a few minutes, then act like it got a power cycle. First things first, check prime power, yup that's good, stable and nothing else plugged it has issues. Next, I open the unit to check the 12V power supply, check, it's good (even monitored the output with another scope for glitches). I start checking other test points and find that during the reset the lower voltage outputs from the mezzanine board are also dropping out. It is unclear if that's normal during a power cycle, or possibly the cause. I investigate for hot components and can't find anything. There have been no error messages prior to a restart, just bam reset.I contacted Tek tech support, they recommend upgrading the firmware (I was running 1.18 just fine) and sure enough during the firmware update, RESET! So now the scope has a discombobulated front panel microcontroller (read: buttons, knobs and lights don't work anymore).Next step, request a service quote for getting it back up and running. Tektronix wants $2710 plus shipping to repair the scope. 5 years old, 1424 hours on it and it's essentially garbage.Has anyone else had experience repairing the original problem of the random resets? Not really sure where to start looking yet. I think that the main acquisition board is ok, when it ran it had no problems getting a signal, and it was nice and clean. Nothing hot that I could tell. Any help out there?

author: darkstar49
author_rank: Frequent Contributor
timestamp: February 04, 2021, 03:51:38 pm
content: Gee... I'd kill the one that advised a firmware update on a scope that does reset randomly !!!!!!!And 3K is the price for an MDO3014 with 3 years warranty (and fresh cal) at a major broker here in Europe... Tek's proposal is plain unacceptable, because even if you have it repaired, you'll still be left with a scope without warranty whatsoever, excepted on the repair.At this stage, you're correct... if you don't manage to fix it yourself, you're left with a piece of electronics garbage...

author: GridWork
author_rank: Contributor
timestamp: February 04, 2021, 07:44:33 pm
content: If I could just get the code back and stable on the front microcontroller, I might be able to troubleshoot what's going on with the rest of the thing. It sure puts the cost of ownership up there. Right now I think it is setting at around $1338 per year. I deliberately went for the Tektronix because I have had many in my professional career that are 20 years old and still functional. Even my TDS210 that was purchased new in 1998 still works.I only did the firmware update because it was running stable for enough time to give me some confidence. The repair cost would have been the same either way.

author: kcbrown
author_rank: Frequent Contributor
timestamp: February 04, 2021, 08:18:10 pm
content: This sounds like very poor design on the part of Tektronix. It shouldneverbe possible to brick an expensive device like this through an incomplete firmware update. It should have a bootstrap algorithm in ROM that performs a checksum verification against the flash and boots from an earlier version of the code that is also in ROM if the checksum fails.The "service manual" for this scope is laughably worthless, not even worthy of its title.

author: darkstar49
author_rank: Frequent Contributor
timestamp: February 05, 2021, 11:27:17 am
content: wasn't there a way to have a 'forced update', even if the scope was unresponsive, by putting some file in the root directory, or renaming the update file ??I can't imagine Tek has to replace h/w because the update went wrong... I'm pretty sure there's a way (although maybe not public) to push an update on a bricked scope...

author: GridWork
author_rank: Contributor
timestamp: February 05, 2021, 02:38:11 pm
content: Since the cost of repair is so high, and fixed regardless of the state of the equipment, I'm going to give it a shot this weekend. I will try and post status updates if I get anywhere.

author: darkstar49
author_rank: Frequent Contributor
timestamp: February 05, 2021, 02:53:31 pm
content: Quote from: GridWork on February 05, 2021, 02:38:11 pmhave you tried putting a file named "forceupgrade.txt" in the root of the USB drive ? (content doesn't matter, can be empty, or maybe must be empty, try both...)

author: GridWork
author_rank: Contributor
timestamp: February 06, 2021, 01:37:35 am
content: Quote from: darkstar49 on February 05, 2021, 02:53:31 pmJust saw that in the firmware readme. I have tried it, and the front panel microcontroller is still shut down. It did get through the firmware with a green check mark at least.

author: GridWork
author_rank: Contributor
timestamp: February 07, 2021, 03:48:03 am
content: Scope is apart for troubleshooting.So far I've been able to see the voltage feeding the CPU board dropout, it mimics the Power ON/OFF button behavior. The firmware on the front panel microcontroller is still AWOL, multiple attempts to force an update have been unsuccessful. I have tracked down the power supply remote ON/OFFThe now brainless microcontroller.I've tried to Telnet/SSH into the scope, but no real responses. The Web interface does work, however it only shows two channels.<EDIT> The screen capture shows all four channels, had to reseat the front end board and all worked ok.

author: darkstar49
author_rank: Frequent Contributor
timestamp: February 07, 2021, 11:24:55 am
content: It won't solve your power issues, but at this stage, I'd already get in touch again with Tek, saying this upgrade was done upon their advice, and that you're now at least at a point where you could have the forced upgrade go through, although without success for the front-panel controller (the Freescale MCU on the 3rd pic). Maybe they could already give you some quote for fixing that part, don't know whether service centres are able to, but there surely is a way (JTAG...) to flash that MCU specifically, without swapping the board.I'm really not a fan of Tek's approach to have a 'dumb' PSU delivering 12V, and provide all other voltages on the mainboard itself... had similar issues on a DPO4034 a while ago, and had to sell it pretty cheap after numerous (and unsuccessful) hours of troubleshooting... (have to say I'm not very at ease with such problems, being more on the software side).

author: m k
author_rank: Super Contributor
timestamp: February 07, 2021, 12:29:42 pm
content: Quote from: GridWork on February 07, 2021, 03:48:03 amWhat else it says, right or wrong model?Is it old or new firmware?

author: GridWork
author_rank: Contributor
timestamp: February 07, 2021, 03:57:01 pm
content: Quote from: darkstar49 on February 07, 2021, 11:24:55 amThe power issue is definitely not from the power supply, it stays rock steady at 12V during the reset event. Having a dumb power supply isn't so bad, it theoretically allows easy future replacement for what is likely to break during normal use. Power lines can get some crazy energies on them.Tek isn't budging on the price to repair much. The only other option I was given was $230 an hour for a technician to troubleshoot + a mandatory $650 calibration fee, not much better. And to add insult to injury, the repair only has a 90 day warranty in either case. <GRRR>

author: GridWork
author_rank: Contributor
timestamp: February 07, 2021, 03:58:08 pm
content: The firmware is whichever version got loaded using the "forceinstall.txt" option.If anyone out there has an MDO3000 series and happens to have a Freescale debug module, would you be willing to download the code from the microcontroller so that I can try and re-flash mine?One of the other things I am noticing is that the quality of the assembly is not up to par for a mid level (and in this price range) oscilloscope. The hand soldered point do not have solder filling the hole, just a light surface fillet on the back side.

author: GridWork
author_rank: Contributor
timestamp: February 08, 2021, 04:16:05 pm
content: New update:I spent most of the weekend trying to trouble shoot what's going on. So far here is what I know:The main application processor is being reset due to a global reset from the supervisor circuit.The supervisor is causing a reset because main voltage is dropping out.The voltage dropout is due to the power controllers on the mezzanine board shutting down.I am trying to route out where the shut down is coming from. The 1.8V controller "RUN" comes from the power good on the 5.0VD controller.It appears that something on the main board tells the AUX DC/DC controllers to start up, not sure where that is yet.I will try and get some pictures posted of the board and attach to this message.

author: kcbrown
author_rank: Frequent Contributor
timestamp: February 08, 2021, 05:19:03 pm
content: Quote from: GridWork on February 08, 2021, 04:16:05 pmThis is actually good news if I'm reading it right. You might not need to re-flash the front panel controller after all, if it's being reset in the way you suggest. Smells like a hardware fault.The very first thing I'd look at is the actual voltages going into and out of the various voltage controllers, see how they vary over time.

author: GridWork
author_rank: Contributor
timestamp: February 08, 2021, 05:58:50 pm
content: Quote from: kcbrown on February 08, 2021, 05:19:03 pmI've been looking at the voltages on all of the DC/DC outputs (and the 12V power supply output), there doesn't appear to be any increased current draw prior to shutdown. There are two dual channel Linear controller chips responsible for the 5.0VD, 3.3V and 1.8V, they are being commanded off from a discrete FET pair (on the backside of the board), but I'm not sure what from. The front panel micro appears to be dead, i.e. no code. It was working prior to the re-flash even when the intermittent reset started.This scope has been sitting on my desk for the last 5 years, and actually un-plugged during a good chunk due to lack of use.One of the really frustrating things is the small components, traces and vias. For as much space as appears on the boards, there is no reason to use the grain of sand parts that they have. 0603's would make for something much more repairable.

author: kcbrown
author_rank: Frequent Contributor
timestamp: February 08, 2021, 11:14:01 pm
content: Quote from: GridWork on February 08, 2021, 05:58:50 pmIt's probably a bill of materials thing. The front-end likely needs the grain-of-sand parts because (I would expect) they have less in the way of parasitic characteristics, and since they're already specifying those parts, they may as well use the same types for the rest of the scope. That would also simplify the pick and place requirements since fewer strips of parts would be needed to feed the pick and place machines.You looked at the DC/DC outputs. What of their inputs? If you can identify which one (if any) is getting a failed input, that might lead you down the right path. If you've got a cracked trace or something, you might be up against a thermally-generated failure and the only way you'd be able to see that is by looking at the inputs right at the voltage controller. But if the voltage regulators are also controlled by a FET pair, and you've already identified the signal from the FET pair as causing the voltage controllers to shut off, then there's nowhere to go except back to the source that feeds the FETs.If you've got a microscope, you should use it to look for cracked solder joints and other types of damage that could be responsible.Sounds like tracing these things out is a real pain.Do you have close-up photos of the area around the FETs (both the side where the FETs are located and the other side)? It might be helpful for us to see what you're up against here ...

author: GridWork
author_rank: Contributor
timestamp: February 09, 2021, 04:06:53 am
content: Quote from: kcbrown on February 08, 2021, 11:14:01 pmPictures inbound:MDO3000 Mezzanine boardTOPBOTTOMLTC3850So what I have found is that the LTC3850 on the Left supplys the 1.8V and 5.0V Digital. The LTC3850 on the right supplies the 5.0V Analog. The LTC3850 on the right PGOOD output feeds the RUN for the one on the left. The 5.0VA (right) is the controller that shuts down first. The RUN1 on it gets commanded off from Q305 (back side, bottom right). Now I've got to figure out what drives it.

author: kcbrown
author_rank: Frequent Contributor
timestamp: February 09, 2021, 05:28:30 am
content: Quote from: GridWork on February 09, 2021, 04:06:53 amIf I'm reading it right, the code on those transistors is I09, which makes them an SST109, an N-channel JFET (seehttps://www.sphere.bc.ca/download/smd-codebook.pdf). The gate is on pin 3, the single pin on the left side in the photo. That connects to R321, which connects to the via just next to the lower left corner of Q305. That via dives through to the other side, but the other side is unpopulated. In fact, the way the other side is unpopulated, and the way these components seem to connect to the unpopulated side, makes me wonder if this is the right place to be looking. This is really weird.Do you have a macro lens or something that you can use to get close-in shots of these two areas?Back side:Front side:I've also attached a horizontally flipped version of the front side. I've tried to match the boundaries and resolution as much as possible so that you can overlay one on top of the other to make it easy to see where any given via goes through and connects to:The problem is that the original images aren't quite high enough in resolution for me to be able to clearly make out the traces and where they go. Additional side lighting from the bottom right might help with that.

author: GridWork
author_rank: Contributor
timestamp: February 09, 2021, 01:16:09 pm
content: Quote from: kcbrown on February 09, 2021, 05:28:30 amI do have a macro and have gotten some more detailed pictures, just ran out of time getting some update here. I think I may have mistyped the transistor, I think that some how Q304 is the one initiating the shut down. It's the only active device I have found connected to the "RUN" pin on the 5.0V DCDC. All of the other components attached to the "RUN" are anodes from various diodes on the board.There may be another development here on the scope. The tech support individual I've been e-mailing mentioned a service note "CS-SUP-0110" about scopes randomly powering down. I have to see if the RMA department changes this from a customer funded repair to something different.

author: gord
author_rank: Newbie
timestamp: February 09, 2021, 03:11:22 pm
content: I have a 5 year old MDO3000 too. It has had very little usage, and in recent weeks I have noticed it resetting. In my case I have only seen this happen during startup. The display is blank when normally the splash screen should be displayed, and it power-cycles, and this happens several times until it manages to fully start up, and it then reports no errors.There was an older firmware release note that discusses a fix which restarts the MDO3000 if the display fails to start up, but I do not know if this is what is happening, or if the MDO3000 software would restart everything if the display does not respond after startup as well. This could all be unrelated and I may have jumped to the wrong conclusion.

author: GridWork
author_rank: Contributor
timestamp: February 09, 2021, 04:54:50 pm
content: Quote from: gord on February 09, 2021, 03:11:22 pmIt sounds similar to what was happening to mine, however I get the splash screen pretty much every time. Sometimes it just seems to hang there, sometimes it will go through the boot. It doesn't show that anything is wrong in the POST results. My issue got compounded when it did one of these resets during an update to the latest (1.30) firmware. That reset corrupted the code in the front panel microcontroller. Even with the latest firmware it still does the restarts.

author: m k
author_rank: Super Contributor
timestamp: February 09, 2021, 07:23:47 pm
content: Is it staying up in the cold?How far back Tek is supporting?Link seems to have MDO3000 Series_v1.10http://twintech21.cafe24.com/v3data/1279

author: GridWork
author_rank: Contributor
timestamp: February 11, 2021, 06:31:11 pm
content: Quote from: m k on February 09, 2021, 07:23:47 pmNot sure what you mean by staying up in the cold? The most recent firmware is v1.30, which didn't fix the problem.

author: m k
author_rank: Super Contributor
timestamp: February 13, 2021, 09:59:54 am
content: If you have a place where the machine can be in low ambient temperature maybe it is not resetting so eagerly.Then if you downgrade far far away back maybe your corrupted part gets some new old code.Assumpton there is that upgrade is upgrading only those parts that are included in that upgrade.Are you back in square one but with new firmware?

author: GridWork
author_rank: Contributor
timestamp: February 15, 2021, 02:36:21 pm
content: Quote from: m k on February 13, 2021, 09:59:54 amHelpful hint: For Tektronix scopes if you create a "forceinstall.txt" file on the USB flash device containing the .img file, then the scope will do an upgrade, regardless if the version is newer or not.I did this rolled it back to 1.18, and the front panel micro is still codeless. Also tried 1.26 and 1.30 (the newest). Makes sense as there should be boot code to tell the microcontroller how to deal with the incoming code. If that got corrupted during the reset, then it wouldn't know what to do. Only the programming port or JTAG port would be able to load new code.I'm still not sure if it's a thermal issue entirely. Since I cannot get the firmware to re-flash the front panel micro, the scope will either get the recycle bin, or sent back to Tektronix to get rejuvenated.

author: m k
author_rank: Super Contributor
timestamp: February 16, 2021, 05:54:27 pm
content: If you like somebody here will buy it from you for sure.I don't know nothing about your specific device but don't let it disturb you.Put device specifics aside for a moment.Do you know what is in that image file, do you know that front panel code is there?It's a good practice to upgrade all but do you know it.I understand that bootloader is a special thing and is not changed during normal write operation.Is the upgrade a script driven, can you edit it?If you can find a spot where front panel is accessed you can add a flag how it is done, like is it skipped or actually written.Maybe it's even saving write cycles and reads first, if that fails there should be a note somewhere.

author: GridWork
author_rank: Contributor
timestamp: February 26, 2021, 01:12:16 pm
content: So to add an update to the MDO saga, the tough decision was made to send the scope back to Tektronix for evaluation. There was apparently no way (for me) to get the code back on the front panel microcontroller.To answer a previous question, the firmware updates are a .img file about 40MB for early versions and 60+MB for later versions. I tried loading several different versions (at least Tek makes them available, not just the latest), none would get the front panel working.Replacing the scope with anything equivalent would end up costing the same or more than the repair, so I'm going to see if Tektronix sees this as a manufacturing defect or customer responsibility.Thanks to all that had suggestions and put some time in trying to help!

author: kcbrown
author_rank: Frequent Contributor
timestamp: March 21, 2021, 11:06:56 pm
content: Quote from: GridWork on February 26, 2021, 01:12:16 pmDid they manage to do anything for you on this?

author: GridWork
author_rank: Contributor
timestamp: March 23, 2021, 12:23:40 pm
content: Quote from: kcbrown on March 21, 2021, 11:06:56 pmThe scope is still back at Tektronix, repairs are being finalized. The end damage was they are replacing the front panel board (I think) for $600, plus another $600 for Calibration (this one is required for any equipment going back) plus the cost of shipping. Half of what was originally quoted, which I can live with even though it makes me grumpy.<Insert Grumpy_Cat meme>I think that the speed things are moving is very much dependent on the times, capacity everywhere seems to be reduced.

author: kcbrown
author_rank: Frequent Contributor
timestamp: March 23, 2021, 06:14:28 pm
content: Quote from: GridWork on March 23, 2021, 12:23:40 pmNormally I would have thought you might want to get a parts unit and perform the replacement from that, but EBay doesn't seem to have any such thing and the used units are generally up in the $5k range from what I can tell. So considering that, $1200 doesn't seem terrible.QuoteYeah, it's not unexpected given the way the virus has been handled.

author: Howardlong
author_rank: Super Contributor
timestamp: June 09, 2024, 10:07:51 pm
content: FWIW, my MDO3014 has started randomly hanging or resetting at boot time over the past few weeks. Occasionally the only way to get it to boot is to pull the AC out, let the caps drain (turn on with the AC unconnected speeds this up).I hadn't used it for quite some time, maybe a couple of years, but occasionally it has the odd feature that is useful and unique on my second bench such as MagniView.I've had it for about 8 or 9 years from new, and it's only done 546 power cycles. Typically it's lived in its Tek carrying bag for most of its life in a dry inside storage room.Once it's up, now it'll stay working perfectly for hours until I finish for the day, but when the problem first started happening it did hang or boot two or three times while in active use. Restoring a setup or setting default setup would do this.Anecdotally, there is a strong correlation between the hanging and the relays clicking away.Also anecdotally, it seems to be getting better rather than worse (touch wood, cross fingers etc).My hypothesis at the moment is that the problem may be due to a relay getting stuck, and that active use has re-wetted the relay contacts.The scope's self test and SPC run OK.

author: Howardlong
author_rank: Super Contributor
timestamp: June 16, 2024, 11:23:03 am
content: Quote from: Howardlong on June 09, 2024, 10:07:51 pmAbout three days ago the scope suddenly completely crashed again while using it, and it failed to ever successfully reboot, stopping at various times during the boot process. After at least half a dozen attempts to get it to boot (turn it off & on, pull the AC cord for a while to let the caps discharge etc), I gave up and put it away in the cupboard to look at later.A day later, with a view to repairing it, I pulled it back out of the cupboard, and it booted fine. It ran a couple of dozen self-tests fine too. However I noticed that the relays don't click when running a self test. Furthermore, if I set up a semi-complex setup to save in an effort to get the relays clicking, apart from setting the 50/75 ohm termination on a channel, the relays won't click... unless you have the TPP analogue probes plugged in. When doing the self test, you are directed to remove all probes, so the relays aren't cycled.I plugged the probes back in, and I haven't been able get it to fail again after 1.5 days of solid use.I now have a new hypothesis... the TekVPI smart probe interface is suffering intermittent connections, leading to a 1-wire comms failure, thus rebooting or crashing the scope. (Previous analysis of these TPP probes show they use a 1-wire solution).So the next time I get a failure and it won't restart, I'll try rebooting with the probes disconnected.

author: Howardlong
author_rank: Super Contributor
timestamp: June 24, 2024, 08:16:01 am
content: Quote from: Howardlong on June 16, 2024, 11:23:03 amI've been using the scope daily for further 8 days, and (cross fingers, touch wood) it's worked flawlessly.Unless it fails again, I'm going to put this down to dirty contacts & intermittent connections on the TekVPI probe connections causing a 1-wire timeout, crashing the scope. This correlates to the failures seemingly coinciding with when the relays on the input attenuator start clicking away.So my advice is, if your MDO3000 starts hanging or crashing randomly, either at boot or while in use, the first step is to remove the probes and see if it works.My unit had been in storage room for many months with the probes removed, so although it was stored inside its Tek bag with its front panel cover on, it's quite conceivable that the TekVPI connections had become a little tarnished, or one or more of the the TPP probes' pogo pin for 1-wire became similarly dirty. I haven't (yet) cleaned the connections, although that will probably be a next step if these problems recur.When on the bench, typically it's set up for days if not weeks or months at a time with the same TPP probes left permanently connected, so it's a little strange this behaviour just suddenly started, but not beyond the realms of possibility.

author: stj
author_rank: Super Contributor
timestamp: June 24, 2024, 12:27:56 pm
content: interesting thread,you should probably dump those 1wire devices and keep a note of their serial number and part id.it's a bit lame though that a 1wire device-search failure didnt just result in a probe message on the screen!

author: Howardlong
author_rank: Super Contributor
timestamp: June 24, 2024, 04:21:16 pm
content: Quote from: stj on June 24, 2024, 12:27:56 pmYes it isn't great if my hypothesis is true, although definitely better than some kind of internal failure!I also have an MDO4000C at my other bench space that's lived out on the bench 24/7 for six years, also with TPP probes. It's not my go to scope on that bench though, it's only had 247 power cycles and 350 hours on, but it enjoys a couple features my daily driver doesn't have (e.g. 60ps MagniVu, USB-HS decode and waveform histograms). I do cycle the probe connections on occasion though with both high speed differential and single ended FET probes, so maybe that has some kind of cleaning effect.

author: northhants
author_rank: Newbie
timestamp: August 12, 2024, 02:44:56 pm
content: Just found that my (work) MDO3024 does not like starting when too warm (like me).It refused to start this morning. It would get as far as the splash page and no further. It had done this occasionally recently, but previously did start on the 2nd or 3rd try. Not this time.But today is also the hottest day so far this year, and I do not have air conditioning in my home office.I also don't have a temperature chamber at home. But the 'scope just fitted in my fridge. After about 20 minutes cooling down, it started OK.

author: Howardlong
author_rank: Super Contributor
timestamp: August 12, 2024, 03:28:04 pm
content: Quote from: northhants on August 12, 2024, 02:44:56 pmI'm sure you may well have already tried this. but did you try booting it without any probes plugged in (cf my problem immediately above)?

author: northhants
author_rank: Newbie
timestamp: August 12, 2024, 05:42:14 pm
content: Quote from: Howardlong on August 12, 2024, 03:28:04 pmSorry - I should have included that I tried the other suggestions first (remove mains (and obviously reconnect!), then remove probes). Still stuck at the into display.Error Log reports 0 system errorsAlso, Powerups: 579Operating time: 18988 hoursJust tried power cycling (I'm stopping now, so doesn't matter if it doesn't start!) - and it started!

author: Howardlong
author_rank: Super Contributor
timestamp: August 13, 2024, 02:01:13 pm
content: Quote from: northhants on August 12, 2024, 05:42:14 pmHas this gone away now (as mine did once I'd figured out I had a probe interface issue), or is it still a problem?Once it's actually started, does it work flawlessly? (In my case, it would crash or reboot anything from a few minutes to a few hours after switching on, as well as on boot).

author: northhants
author_rank: Newbie
timestamp: August 17, 2024, 02:13:24 pm
content: Quote from: Howardlong on August 13, 2024, 02:01:13 pmOnce it started it stayed working. I do not think I have ever seen it reboot after power on.

author: Howardlong
author_rank: Super Contributor
timestamp: August 17, 2024, 06:46:56 pm
content: Quote from: northhants on August 17, 2024, 02:13:24 pmOK, so it only fails at startup, and it still fails at startup (sometimes)? Or have you just left it switched on (understandable!).Yours has about 50x the hours of mine, but only about double the power cycles.