The fisherman say rapidly rising fuel costs are threatening their livelihoods .