Try fixing matrix exclusions

Only test with Django 1.4 on Python 2.x and only test with Django 1.6b on
Python 2.7.x and above.