No, not until your secret passphrase has also been hacked or if an exhaustive search attack can be used to crack your passphrase. Without the other, neither is valuable. Nevertheless, you must deactivate that key and create a new key pair with a new passphrase. You might want to add a new user ID with the information of your new key ID before deactivating your old key so that others are aware of your change of address.