Before revision:  These results suggest that current RNNs do not capture the underlying hierarchical rules of natural language , but rather use shallower heuristics for their predictions .
After revision:  These results suggest that current RNNs do not capture the underlying hierarchical rules of natural language , but rather tend to overfit the training distribution in a way which leads them to perform poorly on `easy' out-of-distribution instances. Thus, while RNNs are powerful models which can pick up non-trivial dependency patterns, inducing them to do so at the level of syntax rather than surface remains a challenge .