fix prettystr stackoverflow issue, and modify test cases

fix this issue

```
local t = {}
t.__index = t
prettystr(t) -- stack overflow
```
Of course this printing of the reference can stay here.