Gradually mix the dirt with kitty litter. You eventually want to be using kitty litter only. Be sure to clean the litter box daily. Cats are naturally clean creatures, and you want to encourage them to use the litter box. Think about introducing a harness or outdoor enclosure.
Cats can be quite content outside on a harness and a leash, basking in the sun.
Outdoor enclosures that are closed on all sides as well as on top are another alternative. If you live in a remote area, you should keep an eye out for your cat as wild animals may attempt to break in. Your crafty cat may try to sneak by you whenever someone opens the door. Keep an eye out. Make sure your cat has permanent identification.