To prevent others from using your Chromecast, go to Account > Devices in the Google Home app and disable Guest mode for your Chromecast.