Zapotek/arachni — Issue #96: Multiple encoding errors during scan
Author: walterl
Created: 2011-11-01 09:58:56+00:00
State: open

I have just started a new scan of the OWASP BWA (with depth=5, link-count=10000) and noticed many encoding errors in the terminal where arachni_rpcd was started:
```...
encoding error : output conversion failed due to conv error, bytes 0x9C 0xC3 0x9D 0x58
I/O error : encoder error
encoding error : output conversion failed due to conv error, bytes 0x89 0x50 0x4E 0x47
I/O error : encoder error
encoding error : output conversion failed due to conv error, bytes 0x9C 0xC3 0xA5 0xC2
I/O error : encoder error
encoding error : output conversion failed due to conv error, bytes 0x9C 0xC3 0xA5 0x59
I/O error : encoder error
I, [2011-11-01T11:16:46.709522 #14463] INFO -- Call: framework.progress_data [127.0.0.1]
I, [2011-11-01T11:16:51.714771 #14463] INFO -- Call: framework.progress_data [127.0.0.1]
I, [2011-11-01T11:16:52.200404 #14461] INFO -- Call: dispatcher.stats [127.0.0.1]
I, [2011-11-01T11:16:52.233533 #14463] INFO -- Call: framework.busy? [127.0.0.1]
I, [2011-11-01T11:16:56.715866 #14463] INFO -- Call: framework.progress_data [127.0.0.1]
encoding error : output conversion failed due to conv error, bytes 0x89 0x50 0x4E 0x47
I/O error : encoder error
encoding error : output conversion failed due to conv error, bytes 0x89 0x50 0x4E 0x47
I/O error : encoder error
I, [2011-11-01T11:17:01.705422 #14463] INFO -- Call: framework.progress_data [127.0.0.1]
I, [2011-11-01T11:17:06.714296 #14463] INFO -- Call: framework.progress_data [127.0.0.1]
I, [2011-11-01T11:17:07.970221 #14461] INFO -- Call: dispatcher.stats [127.0.0.1]
I, [2011-11-01T11:17:08.014558 #14463] INFO -- Call: framework.busy? [127.0.0.1]
I, [2011-11-01T11:17:11.608871 #14461] INFO -- Call: dispatcher.stats [127.0.0.1]
I, [2011-11-01T11:17:11.633084 #14463] INFO -- Call: framework.busy? [127.0.0.1]
I, [2011-11-01T11:17:11.706590 #14463] INFO -- Call: framework.progress_data [127.0.0.1]
I, [2011-11-01T11:17:16.705995 #14463] INFO -- Call: framework.progress_data [127.0.0.1]
I, [2011-11-01T11:17:21.683632 #14463] INFO -- Call: framework.progress_data [127.0.0.1]
I, [2011-11-01T11:17:26.735872 #14463] INFO -- Call: framework.progress_data [127.0.0.1]
I, [2011-11-01T11:17:31.702087 #14463] INFO -- Call: framework.progress_data [127.0.0.1]
I, [2011-11-01T11:17:36.710783 #14463] INFO -- Call: framework.progress_data [127.0.0.1]
encoding error : output conversion failed due to conv error, bytes 0x9C 0xC3 0x8D 0x57
I/O error : encoder error
I, [2011-11-01T11:17:41.708136 #14463] INFO -- Call: framework.progress_data [127.0.0.1]
encoding error : output conversion failed due to conv error, bytes 0x9C 0xC3 0xA5 0xC2
I/O error : encoder error
I, [2011-11-01T11:17:46.712557 #14463] INFO -- Call: framework.progress_data [127.0.0.1]
I, [2011-11-01T11:17:51.711583 #14463] INFO -- Call: framework.progress_data [127.0.0.1]
I, [2011-11-01T11:17:52.199136 #14461] INFO -- Call: dispatcher.stats [127.0.0.1]
I, [2011-11-01T11:17:52.214397 #14463] INFO -- Call: framework.busy? [127.0.0.1]
I, [2011-11-01T11:17:56.780563 #14463] INFO -- Call: framework.progress_data [127.0.0.1]
encoding error : output conversion failed due to conv error, bytes 0x8E 0xC3 0x9B 0x46
I/O error : encoder error
I, [2011-11-01T11:18:01.715101 #14463] INFO -- Call: framework.progress_data [127.0.0.1]
I, [2011-11-01T11:18:06.704651 #14463] INFO -- Call: framework.progress_data [127.0.0.1]
...```

Without any additional information (such as which page(s) caused these errors) determining the precise cause is difficult. The most common causes for such errors are:
1) the encodings declared by page in the XML-header or Content-Type meta tag is not honored by the parser of the page,
2) the page declares one encoding but uses another in its content,
3) the page does not declare an encoding, but contains characters encoded in an encoding other than the parser's default (usually UTF-8)

The particular sequence mentioned in the encoding errors above (0x9C 0xC3 0xA5 0xC2) appears to be either Latin-1-encoded gibberish or two UTF-16-encoded Hangul characters (쎜슥).

These errors did not stop the scanning process, but I thought it might be worth mentioning.

Arachni was run from source, experimental branch at 996460d45df3e60671d58cfd0fd5578a2e8d6212.