[vector-api] hit detect rotated features

This PR fixes a bug where rotated features were not detected by the hit detection code.