This idea is also covered in the Hillis article. The second case occurs more through evolutionary constraints, as similar features are selected for by similar environments and co-evolve. The formal term for this is homoplasy, the opposite of homology. The discrimination between these three outcomes involves an ongoing interplay between the biological analysis of putative homologues and analysis of their phylogenetic implications. Shubin goes on to discuss the proper form for formulation, testing, and explanation of possible theories of homology.
Formulation just requires the scientific skill of observation.
To test a proposition it is to necessarily define it. Shubin implicitly defines it as shared character traits from a common ancestor that are not necessarily similar. Biological homology also cannot be used and is thus not part of the definition of homology. Similarity alone is also a weak test of homology according to Shubin. Ultimately it must be congruent with a robust phylogeny for it to be homology.