rieck/sally — pull request #7 by chwress (2012-08-03 14:49:42+00:00)
Title: strip newline characters at the end of each line of the input file

When processing the input in "lines" mode the newline character at the end of each line remains part of the sample. This behavior comes from the way getline/ gzgetline is defined. However, for correctly processing the input, trailing newline characters need to be stripped. Otherwise, this newline character becomes part of the derived n-grams.

This patch/ branch also reveals another issue with the line-wise processing of the input. Since the input_lines module counts newlines (master) / lines ('num_lines' branch) rather than not empty lines, one cannot skip empty lines when embedding the input in vector space. Otherwise, the expected number of lines and the number of lines actually embedded would differ and Sally stops with a fatal error.