Yes, it's okay to unplug the amplifier from the guitar while playing. However, doing so will result in a much quieter sound as the guitar's pickups will only be picking up the strings' vibrations
This can be useful for practising quietly or playing acoustically, but it won't give you the same volume and tone as playing through an amplifier. Plus, it's important to make sure that you turn down the volume on your guitar before unplugging it from the amplifier to avoid any loud pops or feedback.