Generation of spin squeezing in an ensemble of cold rubidium 87

At the beginning of the 20th century measurement precision was related exclusively to the capabilities of the experimenter and the apparatus. Any imprecision was attributed to imperfect devices for the measurement. Later on, when the quantumness of objects like photons, electrons, or atoms, was recognized, it became clear that the measurement and the object to be measured could no longer be separated. Both follow the laws of quantum physics which impose limits on the accuracy of any measurement. In the last three decades, a lot of research has been dedicated to find strategies to overcome these limitations. This new field of quantum metrology is where this work is placed. This thesis describes the generation of interparticle entanglement in form of a squeezed spin state in a cloud of cold 87Rb atoms by performing nondestructive spin measurements. Towards this goal, we have implemented a shot-noise limited polarization probing and detection system, atom counting by means of absorption imaging, spin state preparation via optical pumping and nondestructive spin state detection. The quantum noise limited spin readout with polarized light is characterized in two complementary ways in terms of sensitivity and all possible quantum and classical noise sources are measured. To address the rich opportunities provided in the multilevel structure of 87Rb with dispersive probing, we have developed and implemented a frequency offsetlock which allows a large tuning range of several GHz for the dispersive probing. Working with a magnetically sensitive atomic system triggered the development of a magnetic field imaging technique. This applies the elongated atomic sample and internal atomic states as a way of measuring magnetic fields over several millimeters with micrometer resolution and sensitivities down to a few tens of pT/ Hz. We developed a new method for tomographic measurements of the atomic density matrix. Based on our ability of shot noise limited detection of polarization rotations we can in principle, characterize the whole atomic spin state. First, measurements are shown and possible improvements are discussed. An in depth characterization of the light-atom interaction between polarized pulses of light and collective atomic states are the basis for measurements at the quantum level. On one side, we apply classical measurements of spin polarized atoms to measure the strength of the lightatom interaction. On the other side, we use quantum measurements of unpolarized atomic states to characterize all noise sources both technical and quantum. One of the major results of this thesis is the realization of quantum nondemolition measurements in a largespin system. We realized limitations to the performance of the QND measurement implied by higher order light shifts in multilevel atoms, which are often neglected for these kind of systems. We develop a technique which suppresses the detrimental effects of higher order light shifts and recovers an ideal QND measurement. The thesis culminates in the demonstration of spin squeezing in the magnetically sensitive ground state of 87Rb. We achieve a quantum noise reduction by 3 dB, where metrological sensitivity is improved by 2 dB beyond the standard quantum limit of a coherent spin state with the same spin polarization. We address the question how spatial and temporal inhomogeneities both in light and atoms influence the preparation of nonclassical states. An existing covariance matrix approach is extended and we apply it to study effects of detector time resolution, spatial inhomogeneities and atomic motion.