Double-Edged Sword

Here I'm going to lay out some of the positions and that come with accepting or denying a species-specific value.
== Speciesm (Or, species-specific value) ==
First, I want to outline that there are multiple versions and implications for speciesm. I'm going to outline which I think are strongest. Then I'm going to outline another position that is somewhat similar to speciesm but is probably a better description of what most people actually are.
1) Species as the only moral criteria.
Here is a position that states that 1a) the only creatures whose considerations matter at all are that of a specific species (humans in our case). One could also slightly alter it to say 1b) The only creatures whose considerations matter are of the species to which I belong
1a) The first interpretation here is to say that humans are important, even if you were, somehow, fundamentally altered to not be human, and therefore lose value. It tends not to be the case that people have moral systems that exclude themselves, but it's also not impossible. Either way, this is unlikely the position people hold.
1b) The second interpretation always guarantees the self as morally relevant, which tends to be more agreeable with most. However, it does exclude pets and it also excludes any other intelligent life, no matter how friendly they are. In general, people aren't overly interested in those consequences.
2) Species + 2a) Pets (or) 2b) Intelligent Life
Here I use an inclusive (or) to signify that choosing 2c) both is an option.
2a) In general, pets are given moral value in the way that they relate to the species with which you belong. So, if you've taken position 1b as a baseline, but agree pets also have value, the value tends to stem from the fact that such an animal has use-value to the species affirmed in 1b. Such uses can be affection or work-related, whether they are representative of certain beliefs (Hinduism), either way, it's how they relate to humans. While this does seem to capture our attitude and seems to account well for cultural relativism, it's also very black and white. Most people here in North America agree dogs are not on the menu and cows and chickens are, still don't tend to treat non-dog animals as "anything goes". I'll address that in the following section: 3).
2b) It seems to be intuitively the case, that if some beings of some sort possessed a certain intelligence and had the capacity to communicate with us, many of us would not wish to eat it. This might not be the case for everyone, but I'd think the majority agrees. While there are many forms of what "intelligent life" is, they tend lead to capacity of communication, civilization-building or perhaps morality building. I've personally set my own bar as self-awareness, the idea that they understand their existence and their mortality.
2c) This is just a combination of holding both positions 2a and 2b at the same time, as additional values, in addition to your own species value. This is a fairly strong non-vegan position.
3) Non-dichotomous moral value.
It isn't required that when you sort different things into different categories that one category says "all rights imaginable" and the other category is "anything goes, it doesn't matter." A lot of people fall into the trap of thinking this might be how moral value must be assigned, but it's not. A non-vegan does not have to hold animals as "worthless", but whatever the value you assign them will be below the threshold of killing and eating for food.
While it's possible to assign everything of your own species a value that exists *above* the threshold of killing and eating for food, how you assign other things can vary. One can hold the position, perhaps, that pets are also above the threshold of killing and eating for food (due to their relation with humans), but they don't hold full human rights, either. One can hold the position that non-pet animals are okay to kill and eat for food, but that just inflicting pain on them to take pleasure in sadism is not okay. When you realize that it is not binary, not "yes" or "no" for value, you can really investigate better how you feel about the moral value of animals. While I do think people widely vary here in dispositions to animals (I have personally heard things like "The suffering makes them taste better" to "Killing and eating is only okay if they are given a very good life"), if it was a gradient, I'd say for most people, non-pet animals tend to have moral value that saves them from pain/torture.
Holding a position of 1b) + 2c) + 3) at the same time is actually very common but most people don't realize that they do, and much of the vegan argumentation technique is to suggest that you can't. This is, however, nonsense.
4) Species alternative.
A similar but slightly different approach is to say "The beings to which I hold connection to are of moral relevance."
That would be to say something like "I am raised by, live with and share a common society with being X. Thus, to me, it holds moral relevance." This seems intuitively plausible. If we had a child raised by wolves, there would be no surprise if they held that wolves were important and relevant in all that person's decision making. Nor would we be surprised if they didn't take much interest in the affairs of humans unless a common dialogue/language bridged that gap. While we, as those who live with and share in human affairs, would see this feral child as something to be saved, helped and improved (due to our relationship with humans), it may look at us as foreign, strange and something to avoid (due to its lack of relationship with humans).
One might find 4) + 2c) + 3) as a stronger position than 1b) + 2c) + 3). It requires you less to hold that species is important, but that one honors relationships they hold.
5) Vegan speciesists
Some might think that being vegan entails that one does not hold a species-specific value, however, this is simply not the case. A vegan can hold the position that all animals are above the threshold of being killed and eaten for food (along with other exploitative* acts). However, in addition to this held belief, is that humans, based on 1a), 1b) or 4) are simply more valuable, in some way or another. It's more important to the vegan position to hold animals to a high standard than it is to exclude additional values.
.* This footnote is just to note that what each person finds exploitative, and what exploitative means, varies.
== The Double-Edged Sword ==
Here is where I draw out the double-edged sword.
 P1) If a vegan holds a species-specific value, then they give legitimacy to holding species-specific values.
P2) If a vegan does not hold a species specific-value, then they hold consequences that are at odds with most of society.
P3) A vegan either holds a species specific-value or they do not hold a species-specific value.
C) A vegan either gives legitimacy to holding species-specific values or hold consequences that are at odds with most of society.
Here, P1, P3 and the following of the conclusion are extremely straight forward. 
P1 is problematic for the vegan position, as you lack the ability to state that speciesism is wrong without being a hypocrite and your view with non-vegans will be reduced to how much value non-animals have.
P2 isn't so problematic for the Moral Objectivist, who, being at odds with society isn't such a big deal to their overall philosophy. Society, for them, is objectively wrong and can be objectively corrected. However, the subjectivist who only finds his expression of value through the use of power and persuasion will find their position highly compromised.
So what makes P2 true?
Here I go for proof by example;
1) Imagine a situation where a baby and a squirrel are in danger. You can only save one. Which do you choose? Immediately you probably recognize that a majority of people would not only pick the baby, but would be disgusted if you chose otherwise.
Now, you may object. Well, the baby is more sentient*, it has a greater life span, it has more capacity for pleasure, people care about it etc etc. You give additional reasons that are not based on species. These are fairly easy to account for however.
...*whatever the hell this means
Imagine a situation where a baby and a squirrel are in danger. The baby has the same sentience, lifespan, capacity and human relations (and whatever other factors you feel like throwing in) as the squirrel. You can only save one. Which do you choose? You'll probably quickly recognize that the equalization of other factors doesn't make it look any better in the eyes of the majority of society. The baby is still the obvious choice over a squirrel to most people. Even some vegans struggle to not just pick the human (thus affirming the antecedent of P1).
 2) If you find that it is okay to have a blind-aid dog, then, an equalized human being who is capable of performing the same task would be a suitable replacement. Naked and leashed (assuming no environmental damage from doing so), this marginal case would be tasked with navigating around the city. Perhaps tie it up outside and go inside a store. None of this would be problematic if one didn't hold an idea of human dignity, a common example of a species-specific value that people hold. Most people would simply reject this conclusion.
3) If it is okay to build houses on land currently occupied by animals (One could reject this by saying humans are not allowed to expand anymore, another unfavorable conclusion), thus displacing the animal's homes and forcing them to relocate or perhaps killing their nests, then it would be equally acceptable to do so to marginal cases who are equalized to said animals. This is another conclusion that people would reject.
For an example of me employing this in a debate:
https://www.youtube.com/watch?v=J2VfxigCNYo&
All of these conclusions have similar problems. We must treat marginal case humans as lesser than other humans if we are to continue our existence. If you state that the results of your own existence and use of resources in this world are justified even if they impact other creatures, then without a species-specific value, you're forced to treat marginal case humans all the same.
It's a double-edged sword because both sides are sharp. Either validate species-specific values or find yourself at odds.