Oops, I wasn't done with that comment.  I was writing:

Or actually, maybe the issue here is that releaseVideo() does not always set restoreTime. How about changing that, so that releaseVideo always sets restoreTime. Either to dom.player.currentTime or to currentVideo.metadata.currentTime or to 0.  Then in restoreVideo you can rely on restoreTime having a meaningful value.  That seems like the cleaner way to do it.