Yes, there is a difference between the middle vowel sound of each word. “Were” rhymes with “err,” while “where” rhymes with “chair.” In other words, you can simply insert an imaginary y-sound in “where,” like saying w-e-y-r, and drop it in “were.”