Fixed serializing/deserializing of closures

1. The upvalue detection could get caught in an infinite
loop in cases when the upvalue was nil.
2. Closures that contained themselves as upvalues caused
another infinite loop that would end in stack overflow.
Such closures are common, as that is how lua implements
recursive local functions.