Q:

Implementation of or in foldr and foldl' in Data. List

Here is the declaration of or function from Prelude
or = foldr (||) False

This implementation is fine if the list fields contains at least one True
If the list contains all False then overflow can happen and  We have to reduce till last element
Here is the implementation using foldl'
or = foldl' (||) False

Here though it is inefficient since it has to reduce till last element of the list irrespective of the contents of the list argument.  But we are guaranteed that we will not encounter an overflow. 
So question is why not use the safer foldl' version

A:

If the list contains all False then […] We have to reduce till last element. 

That is correct.  

If the list contains all False then overflow can happen […]. 

That's not.  Other than functions like (+), (||) isn't strict in both arguments.  It only pattern matches the left one:
(||) :: Bool -> Bool -> Bool
(||) True _ = True
(||) _    x = x

Therefore, if we use the definition of foldr for lists, we get:
foldr (||) False [False,False])
  = False || (foldr (||) False [False]) -- use: False || x = x
  = foldr (||) False [False]
  = False || (foldr (||) False [])      -- use: False || x = x
  = foldr (||) False []
  = False

However, if we use a function where both arguments need to be fully evaluated we get into problems:
foldr (+) 0 [1,2]
  = 0 + (foldr (+) 1 [2])  -- cannot reduce (+), since it needs right hand side
  = 0 + (1 + (foldr (+) 2 []))
  = 0 + (1 + (2))
  = 0 + (3)
  = 3

Which is why you should use foldl' in this case.  But for right-lazy functions like (||) or (:), foldr is perfectly fine.