firedrakeproject/firedrake — pull request #272 by wence- (2014-04-23 09:13:41+00:00)
Title: Assembly of 1-forms in RT-like spaces broken on embedded sphere in parallel

I'm sort of not sure what's going on here. But I tried running Colin's shallow water model in parallel and it broke. It turns out to be a problem building RHSs for the BDM space.

Here's a minimal test:

```python
from firedrake import *

mesh = UnitIcosahedralSphereMesh(refinement_level=0)
global_normal = Expression(("x[0]",
"x[1]",
"x[2]"))
mesh.init_cell_orientations(global_normal)

V = FunctionSpace(mesh,"RT", 1)

u = TrialFunction(V)
w = TestFunction(V)

f = assemble(inner(Constant((1, 1, 1)), w)*dx)

print assemble(inner(f, f)*dx)
```

```
$ python foo.py
5.21571508242
$ mpiexec -n 2 python foo.py
4.68988018723
4.68988018723
```

I have not assured myself that the serial answer is correct, but I claim that it should be the same as the parallel one irrespective.

It's difficult to know quite what's going on here, but one thing that's slightly fishy is that there does not appear to be a bijection between the dofs in serial and those in parallel (once I've sent them through the global to universal numbering):

```python
from firedrake import *

mesh = UnitIcosahedralSphereMesh(refinement_level=0)
global_normal = Expression(("x[0]",
"x[1]",
"x[2]"))
mesh.init_cell_orientations(global_normal)

V = FunctionSpace(mesh,"RT", 1)

u = TrialFunction(V)
w = TestFunction(V)

f = assemble(inner(Constant((1, 1, 1)), w)*dx)

print assemble(inner(f, f)*dx)

import numpy as np
if op2.MPI.comm.size > 1:
gnn2unn = f.function_space()._halo.gnn2unn
vals = np.empty((20, 3), dtype=np.int32)
op2.MPI.comm.Gatherv(gnn2unn[f.cell_node_map().values], vals)
if op2.MPI.comm.rank == 0:
print vals
else:
print f.cell_node_map().values
```

In parallel

```
[[11 10 4]
[12 1 0]
[16 2 3]
[ 3 0 4]
[28 5 1]
[29 6 2]
[23 7 8]
[24 9 8]
[ 7 5 10]
[ 9 6 11]
[27 19 18]
[13 14 12]
[15 16 14]
[18 17 13]
[20 19 15]
[23 22 21]
[24 25 26]
[26 21 27]
[22 28 17]
[25 29 20]]
```

In serial
```
[[ 1 2 0]
[ 3 4 2]
[ 5 6 4]
[ 7 8 6]
[ 9 8 0]
[11 10 3]
[13 12 1]
[14 15 9]
[17 16 7]
[19 18 5]
[22 20 21]
[24 22 23]
[26 24 25]
[28 26 27]
[28 20 29]
[21 13 10]
[23 15 12]
[25 17 14]
[27 19 16]
[29 18 11]]
```

Interestingly, 2-form assembly seems to work. In that the eigenvalues of the serial and parallel operators are the same. Any ideas gratefully appreciated.