improved line processing to remove redundant headers and footers