Get rid of the reference counting code.

The timer function closure already stores that information as a part of
its buffer hash table, use that.