Before you do anything else, steep the whip with the glass connector still attached in some hot water for several minutes. This will allow you to easily remove the connector without breaking it. The connector can then be cleaned with alcohol and q-tips as you would any other glass component.
Do not use alcohol to clean the silicone whip, as it could damage it. Instead use hot water and soap. Rinse the entirety of the pipe until it is clean and then set it aside to fully dry before using it again.