# ENSeg: A Novel Dataset and Method for the Segmentation of Enteric Neuron Cells on Microscopy Images
## 4 Proposed Approach
### Instance Segmentation Fusion  
After the process of generating the candidates and obtaining the user prompts, we can perform the inferences by using SAM-based architectures. To take greater advantage of the method and explore its capabilities, we propose four protocols that refine the final outputs by using both models' outputs. Considering \(I\) as a given input image; \(p\) as an input point prompt; \(c\) as a segmentation candidate, where \(c\in\text{YOLO}(I)\); and \(s\) as SAM's output segmentation mask generated from \(p\) (i.e., \(\text{SAM}(I,p)\)), we propose the following protocols for defining the final segmented object:  
1. Candidate-first: if \(p\) in \(c\), consider \(c\) as the final output and \(s\) otherwise;
2. Prompt refinement: if \(p\) in \(c\), calculate a bounding box from \(c\) and regenerate \(s\). Otherwise, \(s\) is the final output;
3. Segmentation fusion: if \(p\) in \(c\), combine both \(c\) and \(s\) using a fusion rule. Otherwise, \(s\) is the final output;
4. Prompt refinement with segmentation fusion: similar to Protocol 2, but combining the outputted \(c\) and \(s\). Verifying whether \(p\in c\) serves a dual purpose: it functions both as a candidate selection strategy and as a fallback mechanism. Specifically, if YOLO fails to detect or segment a given cell (i.e., \(p\) does not intersect with any detected cells), the SAM model can still be employed to segment it. This approach is particularly beneficial for morphometric and quantitative analyses performed by ENS researchers, as it mitigates false negatives while effectively filtering out false positives. Further details on these protocols are provided later in this section, with corresponding pseudo-codes available in Algorithms 1-4. The first protocol (candidate-first) is described in Algorithm 1. This particular approach allows us to directly evaluate the YOLO's outputs for found candidates. Therefore, it is a direct measure of how the first iteration of segmentation performs, with an increment from the fallback strategy (SAM's outputs). The final segmented objects would have the combination of both \(c\) and \(s\) (when \(p\
otin c\)). ```
1:procedureProtocol1(I, p, c)
2:Input: Image \(I\), point prompt \(p\), segmentation candidate \(c\)
3:Output: Final segmentation result
4:ifp is inside c then
5:return\(c\)
6:else
7:\(s\leftarrow\text{SAM}(I,c)\)
8:\(s\gets s>\text{SAM.mask\_threshold}\)
9:return\(s\)
10:endif
11:endprocedure
```  
**Algorithm 1** Protocol 1--candidate-first. The second protocol (prompt refinement) is described in Algorithm 2. This one uses the bounding boxes from the candidate as input to SAM. SAM's output is then used as the final segmentation mask for the object (when \(p\in c\)). This approach enables the user to interact with the system using a point prompt while having a bounding box performance. According to the authors of [15; 16], point prompts may lead to ambiguity during SAM'ssegmentation process. Therefore, we follow the premise that bounding boxes can reach better segmentation metrics. In this protocol, all objects are segmented with SAM. ```
1:procedureProtocol2(I, p, c)
2:Input: Image \(I\), point prompt \(p\), segmentation candidate \(c\)
3:Output: Final segmentation result
4:ifp is inside c then
5:\(bbox\leftarrow\) calculate_bounding_box(\(c\))
6:\(s\leftarrow\) SAM(\(I,bbox\))
7:else
8:\(s\leftarrow\) SAM(\(I,p\))
9:endif
10:\(s\gets s>\) SAM.mask_threshold
11:return\(s\)
12:endprocedure
```  
**Algorithm 2** Protocol 2--prompt refinement. The third protocol (segmentation fusion) is described in Algorithm 3. Our goal with this protocol is to investigate and understand the possibility of improving the overall performance of those individual models, by complementary objects and masks. To combine two object masks, we perform the boolean operation \(AND\) on the binary masks. It is worth mentioning that the \(AND\) operation works as an intersection, inducing the foreground pixel to only exist when both segmentation masks output such pixels as the foreground.