Well, to stop the flapping, you might consider an elastic strap to hold them in place. That may solve your problem. And you're not in a position to squint and stare at the hand signal - things are happening too fast. I used to have a problem wearing specs under goggles as well.
Most people simply go to contacts.
If you are unwilling or unable, I would recommend you talk to your instructors about letting you jump without goggles. But to do this, you need a pair of specs with large lenses - I recommend the cheap plastic sort. A small lens will not adequately block the wind. When I was on the static line I was able to convince my instructors I would be better off without goggles - and it was a major improvement. When I switched to AFF, they talked me into going back to goggles over the glasses.