Repository: RadioAstronomySoftwareGroup/pyuvdata

Issue:
method to add matched visibilities between 2 objects
e.g. to combine simulated EoR/foreground/noise visibilities.

Needed for pyuvsim: RadioAstronomySoftwareGroup/pyuvsim#28

Discussion:
bhazelton: There are special keywords on github that will close the related issue when this PR is merged if you use them properly. It has to go in the initial (top) PR description and it needs to follow the pattern `closes #___` or `fixes #___`.
bhazelton: I just enabled circleci builds for pull requests from forks (I didn't realize it wasn't enabled until now), so when you push another commit it should actually trigger the circleci builds.
mkolopanis: This will need to be rebased for those github actions to pass now.
mkolopanis: ah the hera_qm failure here is my fault from a QM pr we just merged in. I offloaded a lot of command line arguments to the setup.cfg but that basically requires pytest-cov to run tests. that could be fixed by adding `pytest-cov` to the hera_qm environemnt yaml.

hera_cal is a bit confusing though. It is not downloading `uvtools` anymore via pip? @adampbeardsley do you know if something recently changed in hera_cal that would affect this?
ntk688: Ok I rebased this, and it now has an updated changelog and a brief demonstration in the "adding data" section of the tutorial. 
adampbeardsley: There's this:
https://github.com/HERA-Team/hera_cal/commit/03a1fee8228b21bcf4d0242be524293eabed34d8
But that was three months ago.
The only recent change I know of is that Josh changed the environment to use the most recent pyuvdata. But I don't think that should matter here.
mkolopanis: Ah i think it was a uvtools thing. I made a note of it in #728. The workaround is to manually install basemap because pip is having trouble finding it now? Thanks for digging though.
bhazelton: I'm not sure why the coverage isn't reporting, must have to do with this being a fork, but I can't find a place to fix it in the settings on codecov. I downloaded your branch and ran `test_cover_noinstall.sh` from the scripts folder, which makes output similar to codecov and I found that there are 4 new uncovered lines, line numbers 1464, 1487, 1489 and 1515. The first and last of those will only get called for an inlace sum or diff, the middle two are for a ValueError, we test that Errors are raised as expected using something like this (stolen from another test, you'll need to update for your error):
```    with pytest.raises(ValueError) as cm:
        uvdata_data.uv_object.check()
    assert str(cm.value).startswith('All antennas in ant_1_array must be in antenna_numbers')
```
ntk688: I had some of those error tests, but when I went to investigate I realized those lines weren't being called because of a goofy indenting mistake. So I fixed that and added the in place checks and now test_cover_noinstall.sh shows no uncovered lines in the code I added. Codecov still not going through though...

Patch:
diff --git a/CHANGELOG.md b/CHANGELOG.md
index aa1aabe0..342f2bd4 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -4,6 +4,7 @@ All notable changes to this project will be documented in this file.
 ## [Unreleased]
 
 ### Added
+- `sum_vis` and `diff_vis` for summing or differencing visibilities in the data_array.
 - `read_mwa_corr_fits` for reading in MWA correlator gpubox files and applying cable corrections.
 - `eq_coeffs` for storing equalization coefficients and `remove_eq_coeffs` for removing them.
 - `utils.apply_uvflag` for applying UVFlag objects to UVData objects
diff --git a/docs/tutorial.rst b/docs/tutorial.rst
index 75bcee09..3cf237fe 100644
--- a/docs/tutorial.rst
+++ b/docs/tutorial.rst
@@ -690,7 +690,28 @@ and added to the previous.
   >>> filenames = ['tutorial1.uvfits', 'tutorial2.uvfits', 'tutorial3.uvfits']
   >>> uv.read(filenames)
 
-e) Fast concatenation
+e) Summing and differencing visibilities
+*****************************************
+Simple summing and differencing of visibilities can be done with the ``sum_vis`` and ``diff_vis`` methods.
+::
+
+  >>> from pyuvdata import UVData
+  >>> import copy
+  >>> filename = 'pyuvdata/data/day2_TDEM0003_10s_norx_1src_1spw.uvfits'
+  >>> uv1 = UVData()
+  >>> uv1.read(filename)
+  >>> uv2 = copy.deepcopy(uv1)
+
+  # sum visibilities
+  >>> uv1 = uv1.sum_vis(uv2)
+
+  # diff visibilities
+  >>> uv1 = uv1.diff_vis(uv2)
+
+  # in place option
+  >>> uv1.sum_vis(uv2, inplace=True)
+
+f) Fast concatenation
 *******************************
 As an alternative to the ``__add__`` operation, the ``fast_concat`` method can
 be used. The user specifies a UVData object to combine with the existing one,
diff --git a/pyuvdata/uvdata.py b/pyuvdata/uvdata.py
index 81941090..71ec7c09 100644
--- a/pyuvdata/uvdata.py
+++ b/pyuvdata/uvdata.py
@@ -1424,6 +1424,106 @@ class UVData(UVBase):
             self.check(check_extra=check_extra,
                        run_check_acceptability=run_check_acceptability)
 
+    def sum_vis(self, other, run_check=True, check_extra=True,
+                run_check_acceptability=True, inplace=False, difference=False):
+        """
+        Sums visibilities between two UVData objects.
+
+        Parameters
+        ----------
+        other : UVData object
+            Another UVData object which will be added to self.
+        run_check : bool
+            Option to check for the existence and proper shapes of parameters
+            after combining objects.
+        check_extra : bool
+            Option to check optional parameters as well as required ones.
+        run_check_acceptability : bool
+            Option to check acceptable range of the values of parameters after
+            combining objects.
+        inplace : bool
+            If True, overwrite self as we go, otherwise create a third object
+            as the sum of the two.
+        difference : bool
+            If True, differences the visibilities of the two UVData objects
+            rather than summing them.
+
+        Returns
+        ------
+        UVData Object
+            If inplace parameter is False.
+
+        Raises
+        ------
+        ValueError
+            If other is not a UVData object, or if self and other
+            are not compatible.
+
+        """
+
+        if inplace:
+            this = self
+        else:
+            this = copy.deepcopy(self)
+
+        # Check that both objects are UVData and valid
+        this.check(check_extra=check_extra,
+                   run_check_acceptability=run_check_acceptability)
+        if not issubclass(other.__class__, this.__class__):
+            if not issubclass(this.__class__, other.__class__):
+                raise ValueError('Only UVData (or subclass) objects can be '
+                                 'added to a UVData (or subclass) object')
+        other.check(check_extra=check_extra,
+                    run_check_acceptability=run_check_acceptability)
+
+        # Define the parameters that need to be the same for objects to be
+        # summed or diffed.
+        compatibility_params = list(this.__iter__())
+        compatibility_params.remove('_data_array')
+        compatibility_params.remove('_history')
+
+        # Check each metadata element in compatibility_params
+        for a in compatibility_params:
+            params_match = (getattr(this, a) == getattr(other, a))
+            if not params_match:
+                msg = 'UVParameter ' + \
+                    a[1:] + ' does not match. Cannot combine objects.'
+                raise ValueError(msg)
+
+        # Do the summing / differencing
+        if difference:
+            this.data_array = this.data_array - other.data_array
+            history_update_string = ' Visibilities differenced using pyuvdata.'
+        else:
+            this.data_array = this.data_array + other.data_array
+            history_update_string = ' Visibilities summed using pyuvdata.'
+
+        this.history = uvutils._combine_histories(this.history, other.history)
+        this.history += history_update_string
+
+        # Check final object is self-consistent
+        if run_check:
+            this.check(check_extra=check_extra,
+                       run_check_acceptability=run_check_acceptability)
+
+        if not inplace:
+            return this
+
+    def diff_vis(self, other, run_check=True, check_extra=True,
+                 run_check_acceptability=True,
+                 inplace=False):
+
+        if inplace:
+            self.sum_vis(other, difference=True, run_check=True,
+                         check_extra=check_extra,
+                         run_check_acceptability=run_check_acceptability,
+                         inplace=inplace)
+        else:
+            return self.sum_vis(other, difference=True, run_check=True,
+                                check_extra=check_extra,
+                                run_check_acceptability=run_check_acceptability,
+                                inplace=inplace)
+
     def __add__(self, other, run_check=True, check_extra=True,
                 run_check_acceptability=True, inplace=False):
         """


Test patch:
diff --git a/pyuvdata/tests/test_uvdata.py b/pyuvdata/tests/test_uvdata.py
index 5b55dbe3..67e686db 100644
--- a/pyuvdata/tests/test_uvdata.py
+++ b/pyuvdata/tests/test_uvdata.py
@@ -1642,6 +1642,45 @@ def test_reorder_blts():
     assert str(cm.value).startswith('minor_order can only be one of')
 
 
+@pytest.mark.filterwarnings("ignore:Telescope EVLA is not")
+def test_sum_vis():
+    # check sum_vis
+    uv_full = UVData()
+    testfile = os.path.join(DATA_PATH, 'day2_TDEM0003_10s_norx_1src_1spw.uvfits')
+    uv_full.read_uvfits(testfile)
+
+    uv_half = copy.deepcopy(uv_full)
+    uv_half.data_array = uv_full.data_array / 2
+    uv_summed = uv_half.sum_vis(uv_half)
+
+    assert np.array_equal(uv_summed.data_array, uv_full.data_array)
+    assert uvutils._check_histories(uv_half.history + ' Visibilities summed '
+                                    'using pyuvdata.', uv_summed.history)
+
+    # check diff_vis
+    uv_diffed = uv_full.diff_vis(uv_half)
+
+    assert np.array_equal(uv_diffed.data_array, uv_half.data_array)
+    assert uvutils._check_histories(uv_full.history + ' Visibilities '
+                                    'differenced using pyuvdata.',
+                                    uv_diffed.history)
+
+    # check in place
+    uv_summed.diff_vis(uv_half, inplace=True)
+    assert np.array_equal(uv_summed.data_array, uv_half.data_array)
+
+    # check error messages
+    with pytest.raises(ValueError) as cm:
+        uv_full.sum_vis('foo')
+    assert str(cm.value).startswith('Only UVData (or subclass) objects can be')
+
+    uv_full.instrument = 'foo'
+    with pytest.raises(ValueError) as cm:
+        uv_full.sum_vis(uv_half, inplace=True)
+    assert str(cm.value).startswith('UVParameter instrument '
+                                    'does not match')
+
+
 @pytest.mark.filterwarnings("ignore:Telescope EVLA is not")
 def test_add():
     uv_full = UVData()