The latter is the easiest place to do the bureaucracy because you can anchor in the inner harbor and dinghy over to the customs office. Cruise ships dock in Castries harbor. Most islanders use the cheap minivan bus service to get to town, school or work. Very few buses run on Sunday. If there's no bus stop nearby, you can wave buses down en route as long as there's space for the bus to pull over.
Taxis are plentiful at the airports, in Castries and in the main resort areas.
Always establish the fare with the driver before you get in, doubly so if you want to do anything 'unusual,' like stopping to see a view. There are car rental agencies at the airports and at Rodney Bay. Unless you have an International Driving Permit, you'll need to purchase a local license, which can be picked up from immigration at either airport. If you don't get a license on arrival, most car rental companies will either issue one or take you to a nearby police station to get one. Remember to drive on the left.