The meaning of the Coupling and Invert are the same as on the analog scope. There is also a soft key to turn each channel On or Off, compared to the A, B, or A and B switch on the analog scopes. Two buttons that are new to the HP 54645D 'scope are BW Lim and Vernier. If your input signal is corrupted by high frequency noise, this filter is one way to decrease it. Unlike the analog scopes, it is still calibrated when you use it.
All signals are voltages measured with respect to bench ground.
If you need to measure a voltage difference between two points, and neither of the points can be grounded, then you must measure between one point and ground with channel A1, and between the second point and ground with channel A2. The rightmost softkey should be labeled Probe. You are using a 1x probe without auto-sensing capability, so set it to 1. Using this kind of wiring allows some noise to creep into the signal, so that you can try out the noise-reducing features of the scope. Turn on the function generator.