As soon as you stop it feels a whole lot worse. Other people find that hot water can help. For temporary relief, go to the faucet and turn the hot water on to the point where it is almost uncomfortable. And for four to five hours, you won't itch. Also you should tell everyone the shower should be a very COLD one at the time of infection to close the pores and wash off the poison oak.
Some people find vinegar compresses or rubbing alcohol helps.
I know at least one person who said it will feel better if you pee on it. They are also corticosteriods. Instead of pills the Medrol or Decadron can be given as a shot. This is the route I prefer as I don't have to bother taking pills. I don't know much about poison oak, but I do know that Domboro soaks work good on poison ivy to dry it up.