refactor: unify functions for checking with nameLanguage and that with extensionLanguage when choosing a language parser

getExtensionLanguage was prior to getNamedLanguage. However, we should
use both impartially.

Now ctags can detect shbang line, emacs mode and vim filetype. In many
cases the name of emacs mode or vim filetype is same as the
name of parser.

e.g.

If a file has -*- python -*- line as the head, codes in
getEmacsModeLanguageAtFirstLine will pick up "python" and
choose a language parser getExtensionLanguage.

Though "python" is not registered as an extension.

    extern parserDefinition *PythonParser (void)
    {
	static const char *const extensions[] = { "py", "pyx", "pxd", "pxi" ,"scons",
						  "python2", "python3",
						  NULL };

"python" is the name of language parser.

	parserDefinition *def = parserNew ("Python");

Newly introduced getAnyLanguage checks both
LanguageTable [i]->currentExtensions and  LanguageTable[i]->name
to find a language parser.

Adding the name of parser to the extensions is another approach.
However, we have to change wider area of codes in the approach.

Signed-off-by: Masatake YAMATO <yamato@redhat.com>
refactor: nominate candidates of language parser outside getTwoGramLanguage

In older code getTwoGramLanguage is called after
checkExtensionLanguageCandidates is called.  Though
checkExtensionLanguageCandidates found language parser candidates for
given file extension, getTwoGramLanguage did not use the result;
getTwoGramLanguage found candidates its own code.

In this new code getTwoGramLanguage is called after
nominateLanguageCandidates is called. nominateLanguageCandidates is
replacement of checkExtensionLanguageCandidates.
nominateLanguageCandidates found language parser candidates for given
language specifier(e.g. file extension). getTwoGramLanguage just uses
the candidates.

Signed-off-by: Masatake YAMATO <yamato@redhat.com>
refactor: introduce a function calling both nominateLanguageCandidates and getTwoGramLanguage

Currently language parsers candidates are not considered in
getEmacsModeLanguageAtFirstLine, getInterpreterLanguage,
getEmacsModeLanguageAtEOF and getVimFileTypeLanguage.  We should do
the same in those functions as in file extension handling.

As the first step I make the code for handling file extension generic
enough for applying above functions.

In this patch getSpecLanguage is introduced. getSpecLanguage wraps
sequencial invocations of nominateLanguageCandidates and
getTwoGramLanguage.

Signed-off-by: Masatake YAMATO <yamato@redhat.com>
refactor: use getSpecLanguage when finding a language for emacs mode at the first line

Signed-off-by: Masatake YAMATO <yamato@redhat.com>
refactor: use getSpecLanguage when finding a language for interpreter at shbang

Signed-off-by: Masatake YAMATO <yamato@redhat.com>
refactor: use getSpecLanguage when finding a language for emacs mode at the EOF

Signed-off-by: Masatake YAMATO <yamato@redhat.com>
refactor: use getSpecLanguage when finding a language for vim filetype at the EOF

Signed-off-by: Masatake YAMATO <yamato@redhat.com>
refactor: pass file pointer instead of file name to getTwoGramLanguage

Signed-off-by: Masatake YAMATO <yamato@redhat.com>
add more verbose lines to trace the bahavior of language parser choosing

Signed-off-by: Masatake YAMATO <yamato@redhat.com>
add functions to find a string from stringList

New functions are:

* stringListExtensionFinds
* stringListFileFinds

They will be used in parse.c.

Signed-off-by: Masatake YAMATO <yamato@redhat.com>
store the key for searching TgTableEntry to candidates array

In nominateLanguageCandidates, only LangType was stored to candidates
array of parsers. However, LangType is not enough informative to find
TgTableEntry because a language parser can have multiple TgTableEntry
objects. A TgTableEntry can be find with a key. Here key is assumed as
a file extension, parser name or something string.

To pick proper TgTableEntry for the nominated language parser, not
only LangType but also the key must be stored to candidates array.
This patch introduced parserCandidate strcut type for extending the
candidates array to store the keys for each candidate.

Signed-off-by: Masatake YAMATO <yamato@redhat.com>
deal multiple language candidates in getPatternLanguage

Signed-off-by: Masatake YAMATO <yamato@redhat.com>
add test cases for choosing a parser from multiple candidates nominated from modeline

Signed-off-by: Masatake YAMATO <yamato@redhat.com>
add tgEntry dynamically

This is the initial step for supporting two gram corpus
in regexp languages.

tgEntry was stored to an array. In this patch linked listed
is introduced as the replacement of array. addTgEntry function,
that can add tgEntry to the array, is also introduced.

Signed-off-by: Masatake YAMATO <yamato@redhat.com>
add --<LANG>-corpus option to add a tg table dynamically

Signed-off-by: Masatake YAMATO <yamato@redhat.com>
add a unit test case for testing --<LANG>-corpus option

Signed-off-by: Masatake YAMATO <yamato@redhat.com>
add --list-corpora option

TODO: update ctags.1.
Signed-off-by: Masatake YAMATO <yamato@redhat.com>