Who is Walter talking to?

Rating: 10/10 (14 February 2003)

I've seen this movie about 9000 times, but one line remains a mystery to me. In the scene where Walter tells The Dude that The Jesus is a pederast, the following conversation takes place:
Walter: So how much did they give you? Dude: 20 grand man, and of course I still get to keep the rug. Walter: Just for making the hand-off? Dude: Yeah, they gave the dude a beeper...so whenever these guys call... Walter: What if it's during a game? Dude: Oh, I told them that if it was during leagueplay... Donny: What's during leagueplay? Walter: Life does not stop and start at your convenience, you miserable piece of s***.
My question is about this last line. Who is Walter talking to? It seems unlikely that he's talking to The Dude, because I don't think he would call The Dude a 'miserable piece of s***'. Well, he could, but only when emotions are rising. This seems not to be the case in this scene. On the other hand: if he's talking to Donny (who Ã­s always interfering at his convenience) his timing is way off (his reaction would be way too fast)!
Any ideas?