A successful rule of thumb is three dates.
It's fine to give up if you're not feeling any chemistry or connection. If you want to keep trying, go ahead, but keep in mind his feelings as well.