Removed some panic situations that shouldn't result in panics..

If the goroutines both end, they will both try to close the same things which will cause panics.

Just ignoring panics caused by this specific problem seemed easier than avoiding it altogether.