Tektronix MSO54 repair - Page 1
Forum section: electronics / testgear

author: tmbinc
author_rank: Frequent Contributor
timestamp: September 24, 2025, 11:42:09 pm
content: As I've mentioned over inhttps://www.eevblog.com/forum/buysellwanted/fs(vietnam)-tektronix-mso54-350mhz-4ch-oscilloscope-(used-firmware-issue)/,I recently acquired a defective Tek MSO54 from a fellow eevblogger.The scope safely arrived at my house, and when I booted it, it worked! I played around with it a bit, was very happy, and shut it down. Narrator: But it would never work again*.On every subsequent boot, I was greed with a pretty nondescript generic error message that "a system error has occurred". To dig deeper, I gained root on the embedded Linux (easy as it's not really embedded - it's a standard Ubuntu LTS install on the internal SATA SSD drive), and worked myself through the log files. At some point, what I found was this:[2025-09-11 00:16:09.226] INFO - HWA : Elemental PILOT board ID 137 build ID 1 board rev 7[2025-09-11 00:16:09.226] INFO - HWA : MIA version 138 CvP EEPROM version 138[2025-09-11 00:16:09.230] INFO - HWA : AFG REV D board rev a[2025-09-11 00:16:09.435] INFO - HWA : PMU version string 2.10[2025-09-11 00:16:09.636] INFO - HWA : Found Elemental AFE/RFE FW version 23 in version file ./fe_version.txt[2025-09-11 00:16:11.975] INFO - HWA : AFE board ID 1 board rev 10 firmware version 23[2025-09-11 00:16:11.980] INFO - HWA : AFE board ID 1 board rev 10 firmware version 23[2025-09-11 00:16:12.437] INFO - HWA : TEK049 ICC Cal Version: 1.0.11 ICC Cal Date: Sep 3 2024[2025-09-11 00:16:12.457] INFO - HWA : TEK049 SPI SYS ID 493000[2025-09-11 00:16:12.471] INFO - HWA : TEK049 Serial Number 19021406270401[2025-09-11 00:16:12.665] SEVERE - HWA : Hardware Fault Detected (ICC Fwd Rx BAD Lane Inversion Pattern = ffffffff for Lane 31, between Companion and TEK049)[2025-09-11 00:16:13.401] SEVERE - HWA : Hardware Fault Detected (Companion ICC BAD fwd Rx byte align status=0)[2025-09-11 00:16:13.401] SEVERE - HWA : Hardware Fault Detected (Companion ICC BAD back byte align status=0)[2025-09-11 00:16:13.401] SEVERE - HWA : Hardware Fault Detected (Companion ICC BAD trig info Rx byte align status=0)[2025-09-11 00:16:13.432] SEVERE - HWA : Hardware Fault Detected (ICC byte alignments are BAD!)[2025-09-11 00:16:13.432] SEVERE - HWA : Hardware Fault Detected (Byte alignment failed!)[2025-09-11 00:16:13.432] SEVERE - HWA : Hardware Fault Detected (Companion BAD SYS PLL lock status=1)[2025-09-11 00:16:13.432] SEVERE - HWA : Hardware Fault Detected (Clock lock check failed!)[2025-09-11 00:16:13.432] SEVERE - HWA : Hardware Fault Detected (Companion ICC BAD fwd Rx byte align status=0)[2025-09-11 00:16:13.432] SEVERE - HWA : Hardware Fault Detected (Companion ICC BAD back byte align status=0)[2025-09-11 00:16:13.433] SEVERE - HWA : Hardware Fault Detected (Companion ICC BAD trig info Rx byte align status=0)[2025-09-11 00:16:13.464] SEVERE - HWA : Hardware Fault Detected (ICC byte alignments are BAD!)[2025-09-11 00:16:13.464] SEVERE - HWA : Hardware Fault Detected (Byte alignment failed!)[2025-09-11 00:16:13.471] SEVERE - HWA : Hardware Fault Detected (Companion ICC BAD fwd Rx byte align status=0)[2025-09-11 00:16:13.471] SEVERE - HWA : Hardware Fault Detected (Companion ICC BAD back byte align status=0)[2025-09-11 00:16:13.471] SEVERE - HWA : Hardware Fault Detected (Companion ICC BAD trig info Rx byte align status=0)[2025-09-11 00:16:13.502] SEVERE - HWA : Hardware Fault Detected (ICC byte alignments are BAD!)[2025-09-11 00:16:13.502] SEVERE - HWA : Hardware Fault Detected (Companion ICC Fwd Rx BAD PRBS link status=0)[2025-09-11 00:16:13.503] SEVERE - HWA : Hardware Fault Detected (Companion ICC Back RX BAD PRBS link status=0)[2025-09-11 00:16:13.503] SEVERE - HWA : Hardware Fault Detected (TEK049 ICC Back RX BAD PRBS link status=0)[2025-09-11 00:16:13.503] SEVERE - HWA : Hardware Fault Detected (Companion ICC TrigInfo BAD PRBS link status=0)[2025-09-11 00:16:13.504] SEVERE - HWA : Hardware Fault Detected (ICC PRBS Companion Link Check Failed Lane, Val=0)[2025-09-11 00:16:13.504] SEVERE - HWA : Hardware Fault Detected (ICC PRBS Companion Link Check Failed Lane, Val=ffffffff)[2025-09-11 00:16:13.504] SEVERE - HWA : Hardware Fault Detected (ICC PRBS Companion Link Check Failed Lane, Val=ffffffff)[...]Let's discuss the Tektronix 5 Series architecture a bit. There are some nice teardowns (Signal Path has one) but I've attached a semi-high-res board photo if you want to follow.- Bottom Right there's the clock generation. It's a PLL that feeds of either the external refclk or an oscillator (apparently with the options for more stabilized oscillators as we've seen in the 6 series), and generates a local 50 MHz clock and a 6.25 GHz clock. This fast clock is then distributed via RF amplifiers throughout the board; one for each "Demux" aka. Tek049.- "Demux", aka Tek049, is the Tektronix ASIC that has the 6.25GS/s ADCs. I have the 4 channel version, so there's one of them. The 6 and 8 channel versions have 2 demuxes. The middle slot is unused and, I think, was supposed to be used for the never-released MDO5-series that would have had an RF channel.- "Elemental" is the codename of the 5 Series.- "MIA" is the bridge FPGA that receives the data from the demux, and make it accessible to the embedded PC host over PCIe. Its bitstream is loaded from flash, and "PMU" - a microcontroller - can write to that. MIA is also called "Companion FPGA" or "CvP". There's a "forward", "backward" and "trigger" bus between the Tek049s and FPGA; the forward and trigger bus goes from Tek049 A -> Tek049 RF -> Tek049 B -> FPGA, where the missing Tek049 are "bridged" with these dummy substrates. The backward bus goes the other direction and is clocked from the FPGA. There are 32 lanes for the forward path (since they carry the acquisition data at "low" speed), and 2 lanes for the trigger and back bus. The back bus is used for register access.- AFE is the analog frontend board, I have the 350 MHz oscilloscope so it's the low-end AFE (up to 1GHz); the 2GHz version has another AFE (with the HFD144 amplifier). This is quite similar in design to e.g. the DPO5000-Series (except there the 1GHz version already had the HFD144). Each AFE board has 2 channels.What we can see in the log above is that the general communication appears to work - this is done over a SPI bus - but then the "ICC" - the link between FPGA and Tek049 - fails to train. And we see a "Companion BAD SYS PLL lock status=1".* until we repaired it.[more in the next part]

author: tmbinc
author_rank: Frequent Contributor
timestamp: September 24, 2025, 11:58:43 pm
content: First part is getting it up in a state that allows some debugging. We had a quite sketchy setup that I don't want to explain in detail, but it allowed us to operate the scope via VNC (or rather: see the failure screen via VNC), SSH in to restart the scope process at-will, and at the same time measure what's going on.Initially we looked at the clock generation which all seemed fine. Especially seeing the 6.25Ghz (with an E-field probe and a spectrum analyzer) was a good sign. We noticed that the amplifiers looked a bit different than the more-commonly seen variant from e.g. Signal Path - ours were WW107, whereas the other variant has some other package.We tried ice, we tried heat, we tried pressure - nothing ever changed, the error would be always present. At some point I wrote a Frida script that would repeatedly read the FPGA register to dump the lock state in a loop, but nothing we did would cause it to start working. Sometimes, the PLL locked (per the FPGA register), but the Lane Data was still all-invalid, and very rarely, the scope would continue to boot but no data would be displayed. Apparently, if the right amount of hardware fails to initialize, it will enter a "missing hardware" demo mode.But the error messages showed that the ICC link was bad - so we tried to scope it. The data was non-conclusive, but what we observed was that the BCK (i.e. the FPGA -> Demux) clock was stable at 200 MHz, the FCK and trigger clock (Demux -> FPGA) was.. all mushy, like an unlocked PLL. That was weird, because it was supposed to be a clean clock generated by the Demux. Eventually we noticed that whenever we touched the PCB at one end near the demux, the clock spectrum visibly changed, sometimes to the better! Still not _perfect_, but much better. And at the same time, we noticed a hot spot of two bias resistors that were feeding one of the RF amplifiers (the 6.25GHz is generated by a PLL, split, then amplified, sent to the other side of the board, amplified again, and then split-and-phase-shifted - see Shahriar's analysis, he did a very good job in explaining it). So the last amplifier in the row was a hotspot - A quick on-the-napkin calculation showed that we would expect a ~30mW power draw of the bias resistors in a working state, and about 300mW if the amplifier would be short to ground.Following the 6.25 GHz confirmed the suspicion - the final amplifier was broken and shorted to ground! Likely this was damage from a bit of too much humidity (of which we could see other traces in the instrument - nothing too bad, it looks way worse on the photos on the silver tracks) and ESD? We don't know.However, in a stroke of Tektronix niceness, they populated a few more of the amplifiers than they needed - specifically in the (otherwise unpopulated) RF path, there was another of these amplifiers, completely unused (the passives around it were missing, too). A device that comes with its own spare parts! What a time to be alive.We swapped it in - and it booted nicely! (Just Complained about the AFG missing which we didn't put in yet)

author: TinLethax
author_rank: Regular Contributor
timestamp: September 25, 2025, 12:14:26 am
content: Glad you got it workingCould the extra opamps were used for the paid unlock options 🧐

author: tmbinc
author_rank: Frequent Contributor
timestamp: September 25, 2025, 01:26:37 pm
content: No, the extra amplifiers (not opamps, they are fixed gain wideband amps, similar to an ERA-9SM+ in terms of Package, Pinout, Biasing but with a different marking) were electrically not connected. Even the transmission line was not connected (the DC blocking caps on both input and output were not connected, neither were the bias resistors. You can easily see this in the board.jpg that I've posted, U6603 near the clock gen, and U7203 near the unpopulated middle demux clock distribution.

author: TinLethax
author_rank: Regular Contributor
timestamp: September 26, 2025, 03:12:22 am
content: That's pretty weird. We got spare OPAMP before GTA VI lol

author: Finderbinder
author_rank: Regular Contributor
timestamp: September 26, 2025, 01:45:42 pm
content: Good job done, tmbinc

author: nctnico
author_rank: Super Contributor
timestamp: September 26, 2025, 03:32:52 pm
content: Well done!

author: 2N3055
author_rank: Super Contributor
timestamp: September 26, 2025, 06:47:50 pm
content: Yes, nice creative work and well deserved result. Enjoy your new nice scope!