It could be a marlin, swordfish, big shark or, very frequently, a wahoo. I have found when fishing is slow, the downrigger is often the only thing that produces fish. I never hesitate to drop a bait deep on the downrigger when I meter meatballs of bait hanging deep. It's the only way to get a bait in the strike zone when that's where the feeding is going on. You've been at this game for a long time.
Confidence is as important as technique in fishing.
When getting on the water, you have to assess the conditions and at least start your day in an area you believe in. If it looks good when you get there, if the signs are right, stick to your guns, have confidence in your choice. If you're a charter captain, you have to hope your charter will have confidence in your ability, too. Confidence is born of experience and that only comes with time on the water. I have placed high in numerous marlin tournaments working this area over the years because I have learned it well.