Try to use UBOs to store matrixes for normal map.