Overcome with guilt, he wanted to press the reset button on himself, so that he wouldn't have to live with his actions. But Janeway wouldn't let him - he would have to live with his actions, just like any other member of the crew. It's all part of being flesh and blood, and not just a collection of subroutines. The choices we make in life determine who we are, and we have to live with them, no matter how much we may regret those actions. As a captain, Janeway feels totally responsible for her crew, and their actions.
Leaders always get the credit - and the blame.
That's exactly why Janeway sides with the Hirogen in this episode. The holograms wouldn't be there in the first place if it wasn't for her. Perhaps she's being too defensive, because it was the Hirogen who programmed them with more advanced capabilities. But Janeway wants to control the situation - something that she clearly can't do. She refused to treat Michael as a real person, and thus isolated and punished herself once more, even though she was enjoying herself.