Funny things happen when you scale a vessel down. The result is a cramped and very tender vessel. To build a vessel of a certain size is one thing, but to build a scaled vessel is quite another, and probably a mistake. Fortunately the last forty years of archeological finds has revealed a rich variety of sizes and hull types, and more await discovery. We no longer have to press endless imperfect clones of the Oseberg and Gokstad ships into service for any occasion.
Masts are long, heavy, awkward objects.
The aerodynamic drag of mast and rigging, compared to the bulk of the vessel, is hardly significant in any wind force short of a gale. These three factors mitigate against dropping the mast when under oars. Just because you can, doesn't mean you do it all the time. Not only does dropping the mast not enhance the performance, in a measurable and consistent way, while under oars, but it gives a stern-down trim and clutters the vessel with rigging, spars and spare oars and boathooks. It is useful to drop the mast when pulling the vessel out for the winter, or when passing under low bridges, but if you try to lower it in any sort of a seaway, when it may make some minor difference in performance, it will, more than likely, injure people and damage the vessel.