A chemically amplified negative resist composition for EB or EUV lithography, comprising an alkali-soluble polymer, an acid generator capable of generating an acid acting as a catalyst upon exposure to high energy radiation, and a nitrogen-containing compound as a basic component, and optionally a crosslinker, said polymer as component turning alkali insoluble under the action of the acid acting as catalyst in the presence or absence of a crosslinker, wherein a basic polymer PB comprising recurring units of the general formulae and and having a weight average molecular weight of 1,000 to 50,000 serves as both components and, wherein A is a single bond or a C 1 -C 10 alkylene group which may be separated by an ethereal oxygen atom, R 1 is each independently hydrogen or methyl, R 2 is each independently a C 1 -C 6 alkyl group, B 3 is a straight or branched C 1 -C 10 alkylene group which may contain an ethereal oxygen atom, R 3 and R 4 are each independently hydrogen or a monovalent C 1 -C 10 hydrocarbon group which may contain a heteroatom, with the proviso that R 3 and R 4 are not hydrogen at the same time, R 3 and R 4 may bond together to form a ring with the nitrogen atom to which they are attached, and R 3 and R 4 are a divalent C 2 -C 12 hydrocarbon group which may contain a heteroatom when they form a ring, B 3 may bond with R 3 or R 4 to form a ring with the nitrogen atom to which they are attached, and in this case, the nitrogen-containing ring is a 5 to 7-membered ring which excludes a ring of the structure that a lone pair of the nitrogen atom renders the nitrogen-containing ring aromatic, and the nitrogen-containing ring is not an aromatic ring, a is an integer of 0 to 4, b is a positive integer of 1 to 5, p and q are each independently 0 or 1, t is an integer of 0 to 2, with the proviso that when q=0, the atom in B 3 that bonds with a main chain carbon is an ethereal oxygen atom and B 3 should contain at least two consecutive carbon atoms originating from an alkylene group. wherein A is a single bond or a C 1 -C 10 alkylene group which may be separated by an ethereal oxygen atom, R 1 is each independently hydrogen or methyl, R 2 is each independently a C 1 -C 6 alkyl group, B 3 is a straight or branched C 1 -C 10 alkylene group which may contain an ethereal oxygen atom, R 3 and R 4 are each independently hydrogen or a monovalent C 1 -C 10 hydrocarbon group which may contain a heteroatom, with the proviso that R 3 and R 4 are not hydrogen at the same time, R 3 and R 4 may bond together to form a ring with the nitrogen atom to which they are attached, and R 3 and R 4 are a divalent C 2 -C 12 hydrocarbon group which may contain a heteroatom when they form a ring, B 3 may bond with R 3 or R 4 to form a ring with the nitrogen atom to which they are attached, and in this case, the nitrogen-containing ring is a 5 to 7-membered ring which excludes a ring of the structure that a lone pair of the nitrogen atom renders the nitrogen-containing ring aromatic, and the nitrogen-containing ring is not an aromatic ring, a is an integer of 0 to 4, b is a positive integer of 1 to 5, p and q are each independently 0 or 1, t is an integer of 0 to 2, with the proviso that when q=0, the atom in B 3 that bonds with a main chain carbon is an ethereal oxygen atom and B 3 should contain at least two consecutive carbon atoms originating from an alkylene group.