Fix segfault on device removal

Not sure if this is the correct fix.. the device seems to be getting a POLLERR after it was already deleted. With this patch, it no longer segfaults.