Debugging a breadboard computer: spurious signals, noise, and voltage weirdness? - Page 1
Forum section: electronics / beginners

author: PfhorSlayer
author_rank: Contributor
timestamp: November 27, 2019, 11:55:29 pm
content: Hello all! This is my first post here, and I'm hoping to make this as detailed as possible with all of the issues I'm facing, the things I've tried, and the type of help I'm looking for. Please forgive me if any of this is either too much (useless) information, or the fact that I'm very much learning as I go starts showing through too much. I'll try to split this post up into a few different parts; if the back story isn't helpful, feel free to skip down to the TL;DR.I've built Ben Eater's awesome8-bit breadboard computer, with a few slight modifications. All the pieces work in isolation, but unlike Ben's version (where he uses some sort of black magic or incredible luck), I've had to make a number of modifications to ensure that unused gate inputs are tied (high or low), certain switches are debounced (the ones in his kit and the ones in his videos are different), and things like EEPROM outputs have pull-down resistors to ensure a consistent state while switching addresses. For the most part, though, most of my computer is pretty much the same as what he has designed, circuit-wise. As a brief overview, the computer consists of a clock (with auto and manual modes), a simple ALU that can add and subtract, two 8-bit input registers A and B, a program counter, a cycle counter (each instruction takes 5 cycles), a 4-bit memory address register, 16 bytes of RAM, and an output display module, all connected by an 8-bit bus. The entire computer is built out of 74LS-series ICs, and I've taken photos of my breadboard, which you can seehere. There are alsoschematicsfor all of the modules on Ben's website.Once all the parts are connected together, and I'm starting to try running programs with it, I start seeing lots of glitches, errors, and other problems that I'm not knowledgable or skilled enough (yet!) to resolve myself. I'm a software guy, and this is my firstreal(since I was a small child playing with those 300-in-1 kits) electronics project.The main issue I'm trying to resolve at this time is that the memory address register, which controls the address that is passed to the RAM chips, occasionally resets itself to zero while stepping through a certain instruction cycle. It does not happen every time, and occurs much less often when the register is hooked up to an oscilloscope, but when it does happen it seems to be pretty consistent, always on the same instruction and cycle. It happens on the falling edge of the clock pulse, and the symptoms look as follows: first, the rising edge of the clock pulse pulls the program counter into the memory address register (which I can see happen via the blinkenlights); next, on the falling edge of the clock pulse, the address register is cleared or reset to zero.I have probed all parts of the 74LS173 register chip, and some screenshots of the results are in thephoto album.There is a clear signal, activated via a push-button, that resets the computer's registers and state back to zero. It seems that sometimes, this signal is raised to around 2V when this glitch happens, which resets the register (but ONLY this register, not any of the others connected to the same signal!), despite the button not being pressed. At other times when the glitch happens, I see the Vcc pin for the register going up to ~6.3V, despite the fact that the computer is running at 5V.Here's my main question: how do I go about debugging this kind of thing? When I see a weird signal, or Vcc doing something odd, what kind of steps should I take to try and isolate the problem? It's much easier when the domain is smaller, like when trying to debug an issue with a single module, but when the issue only happens when the entire computer is connected together, I'm presented with such a large problem space that I don't even know where to begin.What kind of debugging methods should I employ in trying to fix these glitches? Are there some steps I can take to reduce the problem space? At the moment (as a software guy), this feels akin to me trying to debug an optimized executable, without symbols, by stepping through instruction-by-instruction. It could be done, but it would be such a monumental undertaking that it would take forever without additional tools. That's what I'm looking for here, additional tools I can use to help me debug these issues.Beyond what's shown in the original schematics, here's what I've already done:Added pull-up resistors to all unused gate inputs; in many cases, I've wired all the unused inputs together, and used a single 1K or 10K resistor to pull the whole set to Vcc.Isolated the edge-detection RC circuit on the clock that's controlling the RAM write from the rest of the clock signal.Changed the auto/manual clock switching 555 timer to be using a monostable configuration, as the switch I have bounces and was causing extra clock cycles if switched at the wrong time. Similar to how the single-step manual clock switch works, the 555 timer remains active while the trigger is high (and is also debounced).Placed a lot of decoupling caps across the power rails, and added a huge 1000nF cap where the power comes in from the wall plug.Added pull-up/down resistors to many outputs where I needed to make sure that the value is never floating. The EEPROM outputs, for example, are all tied low to make sure that they don't float while the EEPROMs switch addresses.Are there other common things I should try? The computer is powered by a 5V, 2A wall plug, and I'm getting roughly 4.8V on all the power rails on each breadboard. This should be enough to handle everything, as far as I can tell.Thank you all for any help or insight you can give me! If (when) additional info is needed, please let me know!TL;DR:I need some guidance on how to go about debugging issues in a large, complex system where all the individual parts seem to work great in isolation.

author: jhpadjustable
author_rank: Frequent Contributor
timestamp: November 28, 2019, 04:36:15 am
content: Welcome! Just a note, local custom encourages you to upload your photos to the forum itself, rather than linking. If imgur goes belly-up tomorrow, then what?QuoteYikes. Power distribution is up to 100% of your problem. I suspect the clear signal is getting pulled high by that power spike, whether by input leakage or otherwise, and doing what it was meant to do. I took a quick look at your photos and here are my notes:1. Real wires are inductive. Each IC package needs to be decoupled from the rail and the rail's inductance. The way you're decoupling that chip, there could be more than two breadboard lengths worth of inductance between the Vcc of the cap and the Vcc of the chip, counteracting the effect of the decoupling cap, never mind the distance between the chip and the power supply, which might be the cause of that current spike. Ideally, the cap should be right at the power pins, or as close as you can reasonably manage, with leads as short as feasible. The sketch below illustrates what should be an effective compromise:It wouldn't hurt to keep a uF or more of electrolytic capacitance every 10-15cm or so along each rail, but see 1a below.1a. Real wires are also resistive. Observe that the top right power rail plug-bar is carrying the entire current of your system, with its inductance and resistance introducing predictable effects on stability of the rest of the circuit's power. I'm only about 51% joking when I suggest building bus bars out of thicker gauge wire (14AWG/1.5mm diameter or so) and soldering stub wires onto them for plugging into the rails.2. 74LS series logic specifies 4.75V minimum Vcc, which must be observedat all times, including transients, to guarantee correct operation. Chargers aren't especially well-regulated because they don't need to be. Try to find a beefier power supply, if you can. Or, choose a logic family more tolerant to lower supply voltages (74HC?).3. TTL output currents are not symmetrical. Especially note that 74LS is only guaranteed to supply 0.4mA when driving high (but 8mA when driving low). A 1k pulldown would dominate that, slowing or stopping the rise into the logic-high voltage range. Use a 4.7k pullup instead, and as a general rule, prefer pullups to pulldowns.Tools... you've got a scope and you've used it effectively to find the problem.Before you make the changes above, you should probe all around the power rails as you reproduce that glitch, so you can get some idea of how wide the effects can be spread. You might also find a logic analyzer to be helpful, maybe not for this sort of problem, but for others that may come to haunt you down the line, especially where you want to record more data points than the scope will allow then review them off-line at your leisure. You don't need an especially expensive one. One of the $20 "Saleae Logic" clones running the FOSS fx2lafw firmware + the FOSS sigrok software would be quite helpful.You may be aware that gdb's step and next commands take an optional numeric parameter. You would probably enjoy having something similar for your build so that you can advance n cycles at will. A counted clock would be fairly easy to build out of a few counters (74LS163), a flip-flop or three (74LS74?), DIP switches to set the cycle count, an AND gate to gate the clock, and a debounced start signal. If you're not too purist about tools, it would be even easier to build with nearly any microcontroller and its built-in timers.

author: tggzzz
author_rank: Super Contributor
timestamp: November 28, 2019, 08:01:24 am
content: That's a well constructed first postjhpadjustable's points are valid.I'll add that you need to consider the (lack of) reproducability and longevity of connections made in solderless breadboards. What works today might not work tomorrow after you have knocked itOther construction techniques are wirewrap (expensive nowadays, good enough for LSTTL), or "manhattan" (good for higher frequencies). But if you want to continue using the computer, consider using a CAD package for the schematic and PCB layout, and getting a PCB made.

author: PfhorSlayer
author_rank: Contributor
timestamp: November 28, 2019, 09:47:55 am
content: Thank you very much, jhpadjustable! I probed around a bit more, and found some areas where Vcc got dangerously close to 4.75V, and a couple places where it dropped under, during the spike. Power is definitely the issue!With your points in mind, I added a second twisted-pair of wires coming from the power plug (and soldered together where they meet at the plug, and crammed in there as tight as possible), and hooked those to the breadboard near where the original two were. I noticed with the single set of wires, that when moving them around slightly while attached to the breadboard, the LEDs would flicker slightly. Definitely not good, but adding the second pair of wires seems to have helped out there quite a bit. I've ordered some header pins, and when they arrive I'll solder both pairs of wires to them and plug it in to the board. Hopefully, that'll be good enough for what I'm doing here!And yep, it was the lack of decoupling capsdirectlyacross the ICs that was causing the resetting issue! I originally figured that just putting them on the rails would be good enough, but as you mentioned, if it's placed next to where the chip's Vcc is attached, the ground lead needs to go all the way down the length of the rail, across the jumper wires to the bottom rail, and back along the length of THAT rail to get back to the ground pin! That's crazy long, and it's now quite obvious why it's not working correctly. I added a small ceramic similar to what you drew out, and sure enough, the resetting problem went away entirely! I'm going to repeat that process for all the other register chips on the board, and as many of the others as I have caps on me to cover. Thank you!As far as the pull-up/pull-down, the only reason I pulled the EEPROM outputs down was because I wanted the "default" state to be low for them, as their signals control the rest of the computer and having them default to high would lead to modules being activated when they should not be, or multiple things trying to output to the bus at the same time, which would definitely not be good!tggzzz: putting it on a proper PCB is definitely the goal, but obviously I need to be sure everything actually does what it's supposed to before I start expanding it (and making it permanent)!

author: tggzzz
author_rank: Super Contributor
timestamp: November 28, 2019, 11:01:33 am
content: Quote from: PfhorSlayer on November 28, 2019, 09:47:55 amYou are now coming to the understanding that you are building ananaloguecircuit, albeit one where chip inputs caninterpretthe analogue input voltages as digital signals. This topic is called "signal integrity", and is very important.Non-registered outputs can cause problems w.r.t. tsusetup times, particularly if they drive a clock input.It is good practice to put decoupling capacitors on all ICs leads, and to ensure you have a very good ground system. Ideally it is a solid ground plane, but for slow stuff like LSTTL you can get away with grids of ground connections.The key calculations you should do are to realise that for an inductor the induced voltage (a.k.a. back EMF) is Vi= Ldi/dt. Rule of thumb is 1mm of wire = 1nH, but what isdi/dtvalue? Assume dt is the outputtransitiontime (not propagation delay), say 5ns for LSTTL. To work out di, start by calculatingthe difference in output current when driving a logic 1 or 0 (1.6mA for LSTTL IIRC?), plusthe current required to charge an input's capacitance. This is i = Cdv/dt, where dv is 2V for LSTTL (2.4-0.4V)and multiply that by the total number inputs connected to the IC's outputs. Strictly you should also include an allowance for switching spikes inside a TTL IC, but the above calculation is sufficient to make the point.Compare that Vi"noise" with the 0.4V headroom on LSTTL inputs in the low state; you will be surprised.QuoteThere's a danger of spending too much time associated with debugging problems inherent in solderless breadboard constructions. Manhattan techniques often have a "tile" for a subcircuit, and when one is debugged it is connected to debugged other tiles. Seehttps://entertaininghacks.wordpress.com/2020/07/22/prototyping-circuits-easy-cheap-fast-reliable-techniques/for some examples. They inherently have good ground planes, see above.

author: jhpadjustable
author_rank: Frequent Contributor
timestamp: November 28, 2019, 11:29:50 am
content: Cheers, glad it all worked out.That's a fair reason for pulldowns. If that's just a sequencer ROM, the few nanoseconds of contention between power-up and the EEPROM output drivers engaging may not be such a big deal. But I'm not especially familiar with the Eater machine, so I won't fight you very hard on that.

author: Ian.M
author_rank: Super Contributor
timestamp: November 28, 2019, 01:33:14 pm
content: jhpadjustable and tggzzz have covered most of the issues, better than I could explain it, but I still have a few comments to contribute.Using bipolar 555 timers without adequate decoupling is asking for trouble. They are notorious for pulling a massive current spike from the supply rail during transition, which will tend to glitch nearby logic due to ground bounce. The first thing I'd try is replacing them with CMOS 555 chips, which are much better behaved and also typically have a true rail to rail output, so will produce cleaner clock edges. They will still need local decoupling, but no worse than a normal logic chip.Daisy-chaining power and ground distribution through multiple breadboards is always asking for trouble. Ground is the most critical - if that's bouncing around too much you don't stand a chance of getting glitch free logic '0' levels at your TTL inputs. Ideally, you'd use something like a copper sheet covering the surface the breadboards are mounted to#, with short wires soldered to it each end of each breadboard to provide low impedance redundant connections to each breadboard's ground strip, but copper foil is expensive, so you may have to settle for a grid of heavy copper wire, running in between and around the breadboards with each horizontal and vertical intersection soldered, (hint: tie each intersection with thin bare copper wire to make them easy to solder). Once you've got a decent ground grid or ground plane, supply wiring is less critical, provided you keep resistance down, and decouple Vcc where it comes onto each breadboard with 10uF electrolytic capacitors*.For power distribution, I'd run a bunch of insulated stranded wires, one for each breadboard, back to a screw terminal block, and solder the breadboard ends to the middle of staple shaped loops of bare copper wire to provide redundant connections to the Vcc strip. The connection to the power supply and to the ground grid should be made at the screw terminal block.# If you don't have all your breadboards fastened to a single board or mounting plate you are fighting connection wobble every time you move them. Replace a chip on one board and you may find a totally unrelated board has developed a bad connection.* To improve decoupling, you'd normally add a 0.1uF ceramic in parallel with each electrolytic to improve high frequency performance, but you should have a ceramic decoupling cap directly across each chip's supply pins anyway, so an extra on a per board basis offers little benefit.

author: tooki
author_rank: Super Contributor
timestamp: November 28, 2019, 01:44:38 pm
content: Quote from: tggzzz on November 28, 2019, 11:01:33 am<semi off-topic>So true.It's something so many people don't understand. One of the things I find so irritating in e.g. audio discussions are people saying that digital cabling will just work, since it's all ones and zeros. I try to remind people that digital doesn't exist at the physical level, it's still analog. Just as you said, it's the abstraction by interpretation of analog signal levels that creates the logical digital information.</ot>

author: tggzzz
author_rank: Super Contributor
timestamp: November 28, 2019, 03:33:54 pm
content: Quote from: tooki on November 28, 2019, 01:44:38 pmWell,audiophilesaudiophools are called that for a reason. Shame they will never have any concept of the steepness of the complementary error function (erfc()) that relates signal to noise ratio to the bit error ratio.

author: bd139
author_rank: Super Contributor
timestamp: November 28, 2019, 03:54:57 pm
content: Good points all round. If I was to build a computer like this from scratch today I'd go straight to board because I'm lazy and that amount of wire wrapping or deadbuggery would give me a panic attack.Don't forget that boards are these days (a) cheap (b) fast and (c) easy to mod / bodge with a scalpel and some kynar and (d) disposable if you socket everything expensive. You can do one run, then fix all the bugs by hand and on the design, then do another one that works.It is actually a massive hat tip to the logic IC designers that the Ben Eater CPU actually works on a breadboard. There's some seriously good noise immunity!

author: PfhorSlayer
author_rank: Contributor
timestamp: November 29, 2019, 04:07:34 am
content: Thank you everyone for the great advice and tips! There's a lot to take in here, so I'm going to spend some time going over everything and doing my best to ensure that what I've got available to me right now (breadboards, and lots of wire!) are as solid as possible. After everything works, I'm planning on expanding it and moving it to proper PCBs (as they're SO DAMN CHEAP now!). Despite the fact that solder-less breadboards have a number of inherent issues, I think by debugging the problems with things laid out like this, where the kinds of issues I'll run into tend to be more pronounced and obvious, will help me learn how to design my PCBs to avoid many such issues, even if they would be far less significant there.Ben Eater's videos and kits are fantastic, but it does seem a littletooconvenient that he was able to make the entire computer work the way he laid it out, with unused gate inputs left floating, absolutely zero decoupling caps across ICs, and everything else hooked up more in line with theory, and less with reality. In fact, I'd wager that every single person who's put together his kits has happened across very similar issues to what I'm facing! It's a great way to be forced to learn about how things work, but it would have been nice to be aware of the potential issues up front, rather than having to dig around asking for help later on.Either way, I'm still learning, and I'm closer to completing the first revision of the board now! Thank you all again for the incredibly helpful information you've all provided!

author: PfhorSlayer
author_rank: Contributor
timestamp: November 29, 2019, 09:58:28 pm
content: As a small update, with the power issues resolved (proper decoupling caps, more lines coming in from the wall-supply to the board, etc.), the computer successfully ran a program to compute the Fibonacci sequence up to 233 (the largest value in the sequence that fits in 8 bits)! Everything works!

author: bd139
author_rank: Super Contributor
timestamp: November 29, 2019, 10:21:48 pm
content: Nice work!