Both Amazon and Google may store voice recordings. While Google claims not to store data unless requested by the user, Google is notorious for putting user privacy last. However, Google is attempting to be more competitive with Apple on this front lately. Both companies state they do encrypt stored data and you can delete it.
Another thing to remember is the skills andapps you use on your smart display may have different privacy policies and may store any voice recordings and interactions on their own servers.
For increased privacy, it's a good idea to cover the camera lens (if there is one) and turn off the mic when the device isn't going to be used.