Set editor mode based on file extension