A punch and die assembly for use in a punch press having an opening for holding the punch assembly, said punch assembly comprising: a guide sleeve adapted to be mounted on the punch press; a punch shank mounted for sliding movement in the guide sleeve and with an insert having a punch point at a lower end thereof for forming or punching a workpiece; said insert comprising a separate removable punch point element having an upwardly extending head portion, said punch shank having a downwardly opening recess therein to receive the head portion of the removable insert, said punch shank having an access port in a sidewall thereof; a manually operable latch element mounted within the punch shank above the insert for engagement with the head of the insert and having at least one laterally movable locking surface positioned to engage a confronting downwardly facing cooperating locking surface as a part of the head of said insert, the latch element having at least one manually moveable operating member that is accessible through the port in the punch shank, the latch element being constructed and arranged on the punch shank to engage the head of the insert such that moving the operating member through the port in the shank will force the latch element by a wedging action to contact the head with a fit so as to eliminate any clearance therebetween, the manually movable operating member being operatively connected to the latch element for pressing the latch element to an insert engaging position so as to retain the head of the insert within the downwardly facing opening in the lower end of the punch shank, characterized in that the latch element is mounted to slide transversely through the punch shank for engagement with the head of the insert, and the latch element comprising a plate having first and second overlapping apertures, the first aperture being of a diameter (R 1) greater than a diameter of said head portion and the second aperture of a diameter (R 2) less than the diameter of the head portion. a guide sleeve adapted to be mounted on the punch press; a punch shank mounted for sliding movement in the guide sleeve and with an insert having a punch point at a lower end thereof for forming or punching a workpiece; said insert comprising a separate removable punch point element having an upwardly extending head portion, said punch shank having a downwardly opening recess therein to receive the head portion of the removable insert, said punch shank having an access port in a sidewall thereof; a manually operable latch element mounted within the punch shank above the insert for engagement with the head of the insert and having at least one laterally movable locking surface positioned to engage a confronting downwardly facing cooperating locking surface as a part of the head of said insert, the latch element having at least one manually moveable operating member that is accessible through the port in the punch shank, the latch element being constructed and arranged on the punch shank to engage the head of the insert such that moving the operating member through the port in the shank will force the latch element by a wedging action to contact the head with a fit so as to eliminate any clearance therebetween, the manually movable operating member being operatively connected to the latch element for pressing the latch element to an insert engaging position so as to retain the head of the insert within the downwardly facing opening in the lower end of the punch shank, characterized in that the latch element is mounted to slide transversely through the punch shank for engagement with the head of the insert, and the latch element comprising a plate having first and second overlapping apertures, the first aperture being of a diameter (R 1) greater than a diameter of said head portion and the second aperture of a diameter (R 2) less than the diameter of the head portion.