runtime: fix parameter checking in syscall.NewCallback

I have made mistake while converting it to Go (CL 132820043).
Added test as penance for my sin.

LGTM=rsc
R=golang-codereviews, bradfitz, rsc
CC=golang-codereviews
https://codereview.appspot.com/136560043