While indexing for some documents there was an execption thrown:

Exception in thread "main" java.lang.IllegalArgumentException: Document contains at least one immense term in field="body" (whose UTF8 encoding is longer than the max length 32766), all of which were skipped.  Please correct the analyzer to not produce such terms.  The prefix of the first immense term is: '[57 57 57 57 57 57 57 57 57 57 57 57 57 57 57 57 57 57 57 57 57 45 45 45 45 45 45 45 45 45]...'

For some reason the wikipedia tokenizer was emitting a token / term of a very large size and the index writer does not like too big terms. I've solved this by limiting the size of the terms via a length filter in the wikipedia analyzer.