Layout detection improvement to detect only a rectangular shape