An apparatus for polishing a peripheral portion of a substrate, said apparatus comprising: a substrate holder configured to hold the substrate and to rotate the substrate; and at least one polishing unit configured to polish the peripheral portion of the substrate, wherein said polishing unit includes a polishing head having a pressing member configured to press a polishing tape against the peripheral portion of the substrate, a tape supply and recovery mechanism configured to supply the polishing tape to said polishing head and to recover the polishing tape from said polishing head, a first moving mechanism (40A, 42A, 43A) configured to move said polishing head in a radial direction of the substrate, and wherein said tape supply and recovery mechanism has guide rollers for supporting the polishing tape, characterised by a second moving mechanism (40B, 42B, 43 B) configured to move said tape supply and recovery mechanism in the radial direction of the substrate, and wherein said guide rollers are arranged such that the polishing tape extends parallel to a tangential direction of the substrate and a polishing surface of the polishing tape is parallel to a surface of the substrate, so that the polishing tape is pressed against the peripheral portion of the substrate from above for polishing a top edge portion. a substrate holder configured to hold the substrate and to rotate the substrate; and at least one polishing unit configured to polish the peripheral portion of the substrate, wherein said polishing unit includes a polishing head having a pressing member configured to press a polishing tape against the peripheral portion of the substrate, a tape supply and recovery mechanism configured to supply the polishing tape to said polishing head and to recover the polishing tape from said polishing head, a first moving mechanism (40A, 42A, 43A) configured to move said polishing head in a radial direction of the substrate, and wherein said tape supply and recovery mechanism has guide rollers for supporting the polishing tape, characterised by a second moving mechanism (40B, 42B, 43 B) configured to move said tape supply and recovery mechanism in the radial direction of the substrate, and wherein said guide rollers are arranged such that the polishing tape extends parallel to a tangential direction of the substrate and a polishing surface of the polishing tape is parallel to a surface of the substrate, so that the polishing tape is pressed against the peripheral portion of the substrate from above for polishing a top edge portion. a polishing head having a pressing member configured to press a polishing tape against the peripheral portion of the substrate, a tape supply and recovery mechanism configured to supply the polishing tape to said polishing head and to recover the polishing tape from said polishing head, a first moving mechanism (40A, 42A, 43A) configured to move said polishing head in a radial direction of the substrate, and wherein said tape supply and recovery mechanism has guide rollers for supporting the polishing tape, characterised by a second moving mechanism (40B, 42B, 43 B) configured to move said tape supply and recovery mechanism in the radial direction of the substrate, and wherein said guide rollers are arranged such that the polishing tape extends parallel to a tangential direction of the substrate and a polishing surface of the polishing tape is parallel to a surface of the substrate, so that the polishing tape is pressed against the peripheral portion of the substrate from above for polishing a top edge portion.