Some of it was, the work you did against friction that went into heating your gears, bearings and even the tires. But a lot of the work didn't go into friction. It went into getting you up the hill, and we say that it got stored in gravitational potential energy. By coasting downhill again, of course. Gravitational potential energy just keeps track of the work you do against gravity in going up a hill.
Potential energy is used to keep track of work done against conservative forces.
At the top, the mass is at rest, so its kinetic energy K is zero. As the mass swings down, it picks up as much kinetic energy as the potential energy it loses. One can therefore calculate K at any height. 2gh Note that the speed doesn't depend on mass. A big kid has just the same speed as a small kid if released from the same height.