Be more strict about excluding characters

After a closer inspection of Graphite's behavior, a more complex
approach is needed.