No, you should not be worried if someone knows your public IP address. Your public IP address is shared with every site that you visit, and it is used to identify your device on the internet. Other individuals and online platforms will have access to it when you interact with them.