Fixed debugger toggle issue causing breakpoints to show up

Things seem frail right now, but I haven't uncovered any other errors.
Hopefully everything is stable now though.