Repository: ericsson/codechecker
Issue: Hash generation failed [ERROR 2018-11-05 14:44] - 'utf8' codec can't decode byte 0

Xerces 3.1.4 sensitive profile store fails.


A clear and concise description of what the bug is.
[INFO 2018-11-05 14:43] - [1255/1258] clangsa analyzed SGXMLScanner.cpp successfully.
[INFO 2018-11-05 14:43] - [1256/1258] clangsa analyzed PSVIWriterHandlers.cpp successfully.
[INFO 2018-11-05 14:44] - [1257/1258] clangsa analyzed TraverseSchema.cpp successfully.
[INFO 2018-11-05 14:44] - [1258/1258] clangsa analyzed TraverseSchema.cpp successfully.
[INFO 2018-11-05 14:44] - ----==== Summary ====----
[INFO 2018-11-05 14:44] - Total analyzed compilation commands: 1258
[INFO 2018-11-05 14:44] - Successfully analyzed
[INFO 2018-11-05 14:44] -   clangsa: 629
[INFO 2018-11-05 14:44] -   clang-tidy: 629
[INFO 2018-11-05 14:44] - ----=================----
[INFO 2018-11-05 14:44] - Analysis finished.
[INFO 2018-11-05 14:44] - To view results in the terminal use the "CodeChecker parse" command.
[INFO 2018-11-05 14:44] - To store results use the "CodeChecker store" command.
[INFO 2018-11-05 14:44] - See --help and the user guide for further options about parsing and storing the reports.
[INFO 2018-11-05 14:44] - ----=================----
[INFO 2018-11-05 14:44] - Analysis length: 415.593192101 sec.
[INFO 2018-11-05 14:44] - Storing analysis results for run 'Xerces_3.1.4'
[ERROR 2018-11-05 14:44] - Hash generation failed
[ERROR 2018-11-05 14:44] - 'utf8' codec can't decode byte 0xb7 in position 2475: invalid start byte
[ERROR 2018-11-05 14:44] - Hash generation failed
[ERROR 2018-11-05 14:44] - 'utf8' codec can't decode byte 0xb7 in position 2475: invalid start byte
[ERROR 2018-11-05 14:44] - Hash generation failed
[ERROR 2018-11-05 14:44] - 'utf8' codec can't decode byte 0xd7 in position 1680: invalid continuation byte
[ERROR 2018-11-05 14:44] - Hash generation failed
[ERROR 2018-11-05 14:44] - 'utf8' codec can't decode byte 0xd7 in position 1680: invalid continuation byte
[ERROR 2018-11-05 14:45] - API call error: massStoreRun
[ERROR 2018-11-05 14:45] - RequestFailed(errorCode=2, extraInfo=None, _message="'utf8' codec can't decode byte 0xb7 in position 2475: invalid start byte")
3.1.4 failed to store

Changed files:
libcodechecker/report.py, libcodechecker/util.py

diff --git a/libcodechecker/report.py b/libcodechecker/report.py
index 923a961a..65984619 100644
--- a/libcodechecker/report.py
+++ b/libcodechecker/report.py
@@ -86,7 +86,9 @@ def generate_report_hash(path, source_file, check_name):
         from_col = m_loc.get('col')
         until_col = m_loc.get('col')
 
-        line_content = get_line(source_file, source_line)
+        # WARNING!!! Changing the error handling type for encoding errors
+        # can influence the hash content!
+        line_content = get_line(source_file, source_line, errors='ignore')
 
         if line_content == '' and not os.path.isfile(source_file):
             LOG.error("Failed to generate report hash.")
diff --git a/libcodechecker/util.py b/libcodechecker/util.py
index 46f3b97e..9f4aef09 100644
--- a/libcodechecker/util.py
+++ b/libcodechecker/util.py
@@ -490,21 +490,31 @@ def escape_like(string, escape_char='*'):
                  .replace('_', escape_char + '_')
 
 
-def get_line(file_name, line_no):
+def get_line(file_name, line_no, errors='replace'):
     """
     Return the given line from the file. If line_no is larger than the number
     of lines in the file then empty string returns.
     If the file can't be opened for read, the function also returns empty
     string.
+
+    Try to encode every file as utf-8 to read the line content do not depend
+    on the platform settings. By default locale.getpreferredencoding() is used
+    which depends on the platform.
+    Encoding errors are handled by replacing the character with '?'.
+
+    Changing the encoding error can influence the hash content!
     """
     try:
-        with io.open(file_name) as source_file:
+        with io.open(file_name, mode='r',
+                     encoding='utf-8',
+                     errors=errors) as source_file:
             for line in source_file:
                 line_no -= 1
                 if line_no == 0:
                     return line
             return ''
     except IOError:
+        LOG.error("Failed to open file %s", file_name)
         return ''