Custom brushes should be saved as .ABR files which is the default file format used by Photoshop for brush presets.