This is not quite right yet, but very very close. Right now your shutdown logic is racy. You call stop, which notifies, but then you don't wait before tearing down the socket in the TCPServer::stop method. But you don't know that the running thread has actually unblocked from select yet.

Maybe consider making TCPServer own its thread? Then TCPServer::start creates the thread, and TCPServer::stop does the following:

// notify the wakeup pipe
// join the thread, blocking until it stops running
// teardown the sockets