Actually reload config when doing reload()