Q:

What sort of check should be used for my frenzy ability's stay-in-control aspect? 

My D&D 3.5e character is a half-shark/half-man, and I want to give him a boost ability that allows him to indulge in his frenzy or instinct and gain bonuses during battle.  Doing so, however, would run him the chance of losing control and attacking anything, friend or foe.  What sort of check would make the most sense as far as maintaining control of/snapping out of the frenzy?  A Will Check that gets harder every round, INT vs WIS check, or maybe a Concentration check? 
I was looking to create a sort of boost ability for my character since we need another melee fighter and our strongest characters seem to have exactly those.  Of course, their boosts cause them some major fatigue after using them and I wanted to try and find a way around that because we've almost been tossed the excrement end of the stick coming out of a tournament or a major fight, and unable to defend ourselves or barely run away from whatever relatively easy troubles happen to us right after.  But I suppose a half shark would bring some great non-combat skills too 

A:

One, most tables really hate this kind of thing
These kinds of effects tend to ruin, or severely sidetrack, campaigns, to the point that the rampaging character makes it impossible to actually make any kind of progress.  Make absolutely certain that every single player in the game is on board with this.  It is not enough, in this case, to just check with the DM; just because the DM is OK with it does not necessarily mean your fellow players are, and every one of those players has an expectation and right to enjoying the game.  You don't have a right to ruin that enjoyment, which this character very well might. 
We get very many questions from DMs trying to prevent this kind of thing, because it's messing up the game for everyone. 
Two, you can create a character like this without homebrew
Just so you are aware, the Frenzied Berserker from Complete Warrior behaves like this.  Which is a reason why it's frequently banned.  But if it's not, and the group is 100% on board with this character, taking levels in Frenzied Berserker is a good way to model this character.  It provides fairly substantial bonuses, and reasonably-thorough rules on how the frenzy is triggered and how the berserker behaves while frenzied. 
Three, you can and should base your homebrew on the Frenzied Berserker
The relevant rules for the Frenzied Berserker are below. 

The character can enter a frenzy as a free action. [...]  In addition, if she takes damage from an attack, spell, trap, or any other source, she automatically enters a frenzy at the start of her next action, as long as she still has at least one daily usage of the ability left.  To avoid entering a frenzy in response to a provoking effect, the character must make a successful Will save (DC 10 + points of damage taken since her last action) at the start of her next turn. 
While frenzied, the character cannot use any Charisma-, Dexterity-, or Intelligence-based skills (except for Intimidate), the Concentration skill, or any abilities that require patience or concentration, nor can she cast spells, drink potions, activate magic items, or read scrolls.  She can use any feat she has except Combat Expertise, item creation feats, or metamagic feats. [...] 
Should she run out of enemies before her frenzy expires, her rampage continues.  She must then attack the nearest creature (determine randomly if several potential foes are equidistant) and fight that opponent without regard to friendship, innocence, or health (the target's or her own). 

Here, you see that it's a Will save to avoid going into an unintentional frenzy, and once frenzied it just lasts a certain duration.  The bonuses from the frenzy (not quoted) are fairly substantial, so that (arguably) makes some sense.  In your case, it may not (probably should not as a racial ability), so probably 1/round make a Will save to end the frenzy.