I’ve had this issue before, so I have some ideas. For instance, if you can’t find matching mortar or there isn’t any available at your local home improvement store, you can try using some masonry glue or cement grout instead. If you do decide to use this method, make sure that you seal it well with some paint or stain sealer after it has dried completely. This will prevent it from being discolored by moisture or water damage. Another idea would be to apply some grout tinting agents that are specifically designed for brick grout. You could also use colored mortar instead of coloring agents, but this will take some work on your part to find matching mortar colors. It may also require some patience because you will need to apply multiple coats to make sure it’s as uniform as possible.