abstract: Phagocytosis is the fundamental cellular process by which eukaryotic cells bind and engulf particles by their cell membrane. Particle engulfment involves particle recognition by cell-surface receptors, signaling and remodeling of the actin cytoskeleton to guide the membrane around the particle in a zipper-like fashion. Despite the signaling complexity, phagocytosis also depends strongly on biophysical parameters, such as particle shape, and the need for actin-driven force generation remains poorly understood. Here, we propose a novel, three-dimensional and stochastic biophysical model of phagocytosis, and study the engulfment of particles of various sizes and shapes, including spiral and rod-shaped particles reminiscent of bacteria. Highly curved shapes are not taken up, in line with recent experimental results. Furthermore, we surprisingly find that even without actin-driven force generation, engulfment proceeds in a large regime of parameter values, albeit more slowly and with highly variable phagocytic cups. We experimentally confirm these predictions using fibroblasts, transfected with immunoreceptor Fc$\gamma$RIIa for engulfment of immunoglobulin G-opsonized particles. Specifically, we compare the wild-type receptor with a mutant receptor, unable to signal to the actin cytoskeleton. Based on the reconstruction of phagocytic cups from imaging data, we indeed show that cells are able to engulf small particles even without support from biological actin-driven processes. This suggests that biochemical pathways render the evolutionary ancient process of phagocytic highly robust, allowing cells to engulf even very large particles. The particle-shape dependence of phagocytosis makes a systematic investigation of host-pathogen interactions and an efficient design of a vehicle for drug delivery possible.
author: Sylvain Tollis$^{1, 2,}$, Anna. E. Dart$^{2, 3}$, George Tzircotis$^{2, 3}$, and Robert G. Endres$^{1, 2, *}$
title: The zipper mechanism in phagocytosis: energetic requirements and variability in phagocytic cup shape

# Background

Phagocytosis is the ancient, evolutionarily conserved process by which eukaryotic cells bind, engulf, and destroy particles and cells larger than 0.5$\mu$m in diameter . The importance of phagocytosis is derived from its two main functions: (1) a feeding mechanism in single-cell organisms , and (2) the clearance of pathogens, apoptotic and senescent cells from our body by immune cells . As part of our immune defense, phagocytosis is mainly performed by professional phagocytes, including macrophages, neutrophils, and dendritic cells. Initiation of phagocytosis occurs with recognition of the target particle either directly or via an opsonising molecule. For instance the Fc portion of immunoglobulin G (IgG) is recognized by the cell-surface receptor Fc$\gamma$RIIa . Ligand-receptor binding triggers intracellular signaling , resulting in remodeling of the actin cytoskeleton and coherent growth of cell membrane around the particle to form the phagocytic cup . Eventually, the leading edge of the growing cup closes, and a membrane vesicle enclosing the particle (phagosome) moves inside the cell. Subsequently, the phagosome fuses with vesicles containing enzymes , acids , and oxygen radicals to destroy the particle.
The biochemical pathways involved in phagocytosis are complex. Dozens of cell-surface receptors contribute to the recognition of a large variety of ligand molecules and subsequent particle engulfment . The Fc$\gamma$ receptor (Fc$\gamma$R) and complement receptor 3 (CR3) of the integrin receptor family are the most widely studied and understood receptors involved in phagocytosis. Fc$\gamma$R-mediated phagocytosis proceeds through membrane protrusions and leads to thin cups , whereas in CR3-mediated phagocytosis, particles appear to sink into the cell . Spreading of the cell membrane over the particle involves actin-driven cell-shape changes similar to the processes involved in cell migration and adhesion . Specifically for Fc$\gamma$R, binding to an IgG-opsonized particle results in receptor clustering and recruitment of small GTPases of the Rho family, which, via proteins of the WASP family, activate the Arp2/3 complex . The latter promotes branching of actin filaments, leading to an increase in the number of uncapped ends and to an isotropic growth of the actin network . Additionally, the phagocytic cup has been shown to be enriched in gelsolin , coronin , and other regulators of actin polymerization. All in all, this complex signaling pathway involves 100-1000 different types of molecules , rendering mathematical modeling at the molecular level impossible.
Despite the huge biochemical complexity, the engulfment process shows a strong dependence on simple biophysical parameters. First, it relies on the availability of extra membrane at the phagocytic cup , provided by delivery of membrane vesicles or unwrinkling of membrane folds . Second, completion of phagocytic uptake depends on the shape of the particle and, interestingly, on the initial orientation of the particle on the cell surface . For instance, experiments demonstrate that elongated spheroid polystyrene particles coated with IgG are more efficiently engulfed when presented to the phagocyte with their tip first. Third, a recent study by one of the authors demonstrates that the biophysical requirements for phagocytosis lead to either complete phagocytosis or stalled cups due to the presence of a mechanical bottleneck . Interestingly, the same study shows that engulfment appears to even proceed in cells treated with (modest amounts of) cytochalasin D, an inhibitor of actin polymerization, indicating that biochemical pathways may not always be necessary for this initial stage of phagocytosis.
The mechanism of phagocytosis is only partially understood, with key insights provided more than three decades ago. In the 1970's, Griffin and his collaborators demonstrated that incomplete coating of particles with ligand results in only partial uptake. This indicated that phagocytic uptake occurs via successive zipper-like ligand-receptor binding (Figure 1A), and not by an all-or-nothing mechanism triggered at the onset of phagocytosis. The zipper mechanism is the underlying assumption in a number of recent modeling works in phagocytosis and endocytosis , mainly addressing the influence of the cell-membrane tension and ligand-receptor bond density on engulfment. Despite the general acceptance of the zipper mechanism, many of its biophysical requirements are insufficiently understood. Questions, so far unanswered, include what the energetic requirements of the zipper mechanism are, specifically what role actin polymerization plays in its progression during phagocytosis, and also whether the zipper mechanism can explain the particle-shape dependence of phagocytosis. Previous models were unable to fully address the particle-shape dependence, as they assume rotational symmetry around the axis connecting cell and particle. Additionally, large particle-to-particle variation in cup growth and cell-to-cell variation in the related process of endocytosis point towards the importance of stochasticity during the uptake, not captured in previous deterministic approaches.
Recent experiments provide new insights into the biophysical mechanism for driving the membrane around the particle, suggesting a ratchet-type mechanism. Once started, phagocytosis progresses unidirectionally and irreversibly . This irreversible membrane progression is further supported by the loss of lipid and protein mobility at the phagocytic cup, observed using fluorescence recovery after photobleaching (FRAP) . While several models proposed mechanisms of force generation by actin polymerization (see and references therein), recent experiments based on fluorescent speckle microscopy demonstrate that actin does not directly push the membrane outwards. Instead, by filling gaps provided by membrane fluctuations (or other types of membrane movement), actin polymerization prevents the membrane from moving backwards like a ratchet . The relevance of such Brownian ratchets in biology has previously been emphasized . The question is if a ratchet mechanism, together with energetic restrictions in membrane bending and stretching, can naturally lead to phagocytic uptake and account for the shape-dependence of phagocytosis.

In this work, we propose a ratchet-like biophysical model for the zipper mechanism. This model differs from previous works in that it is, to our knowledge, the first fully three-dimensional stochastic model of phagocytic engulfment. Specifically, thermal membrane fluctuations, assumed to play a major role in our model, provide the energy source to locally deform the membrane and to build further ligand-receptor bonds for zippering the membrane around the particle. Actin polymerization makes ligand-receptor bonds effectively *irreversible*, *i.e.* reinforced and stabilized for a significant amount of time. To investigate the role of actin, we compare cup progression for the regular *active* zipper with a *passive* zipper model in which ligand-receptor binding remains specific and strong but *reversible* due to the absence of actin polymerization.
Interestingly, we find that the passive zipper also leads to engulfment of small particles, rendering phagocytosis highly robust. However, such passive engulfment is generally slower and produces much more variable phagocytic cups than the active zipper. Furthermore, our computer simulations lead to successful phagocytic engulfment in a broad range of parameters values, including different particle sizes. For non-spherical particles, completion of engulfment depends strongly on particle shape and orientation. Our model further predicts that cup shape invariably depends on membrane biophysical parameters, in particular surface tension and cell-volume constraint.
To test the predicted difference between the active and passive zippers, we experimentally implement the two different types of zippers using COS-7 fibroblasts which, after transfection with GFP-tagged Fc$\gamma$ receptor, phagocytoze IgG-coated polystyrene particles. Specifically, we performed phagocytic assays under three different *conditions*: (1) cells expressing wild-type Fc$\gamma$R for the active zipper (WT-Fc$\gamma$R), (2) cells expressing a signaling-dead mutant receptor (Y282F/Y298F-Fc$\gamma$R), which specifically binds IgG ligand but is unable to signal to the actin cytoskeleton , and (3) cells expressing WT-Fc$\gamma$R and treated with cytochalasin D (WT-Fc$\gamma$R$+$CytoD). The last two conditions represent two versions of the passive zipper due to the absence of actin polymerization in phagocytic cups. To compare with our model, we systematically analyze confocal microscopy images, and quantitatively estimate cup variability for the three different conditions using *small* (1.5 $\mu m$ radius) and *large* (3 $\mu$m radius) particles. Consistently with our simulations, phagocytic cups develop more slowly and are significantly more variable in the absence of actin polymerization. Our results provide new insights into the robustness of phagocytosis, as well as the role of bacterial cell shape in host-pathogen interactions.

# Results and Discussion

## Ratchet model for the zipper mechanism

Our model is based on the following experimental observations. Engulfment of quasi-spherical particles by neutrophils progresses continuously without significant pause or reversal, indicating that ligand-receptor binding is essentially irreversible . This irreversibility is further supported by FRAP and single-molecule experiments, which show that lipids and proteins in phagocytic cups, as well as ligand-bound Fc-receptors are immobilized in an actin-dependent fashion . Additional support for the notion of irreversible uptake was recently determined in a related context . Fluorescent speckle microscopy of actin flow and image analysis during cell migration show that the membrane at the leading edge protrudes first, followed by actin polymerization to fill the gap between the membrane and the actin cortex. Such actin polymerization is mainly restricted to the leading edge due to signaling by receptors and/or localization of small GTPases of the Rho family . The role of actin polymerization in phagocytosis is hence to stabilize ligand-receptor bonds and to rectify membrane movements in a ratchet-like fashion, leading to unidirectional movement of the leading edge of the engulfing cell.
Figure 1A introduces the general concept of the zipper mechanism in phagocytosis, and Figure 1B summarizes our ratchet model for this mechanism. The cell membrane and the actin cortex are described by a Helfrich-type energy function , including contributions from ligand-receptor binding, membrane bending and stretching, as well as a cell-volume constraint. Chosen membrane parameters effectively describe the cell plasma membrane with its underlying actin cortex. The model was implemented using finite-temperature Monte Carlo simulations of the discretized cell membrane (see *Methods* for details). Briefly, the algorithm proposes random, thermally generated membrane fluctuations (trial moves), which are either accepted or rejected depending on the change in total energy during the move. When a random membrane fluctuation brings the cell membrane in contact with the particle, this fluctuation is likely accepted due to energetically favorable ligand-receptor binding. Once accepted, this fluctuation is made irreversible as a result of signaling and actin polymerization. In contrast, a membrane fluctuation far away from the particle is less likely to be accepted. Even if accepted, the fluctuation is not made irreversible and hence may retract at a later time (see Supplementary Fig. S1). Hence, in our model the actin network only supports membrane fluctuations which lead to progression of the engulfing zipper as a result of signaling.

## Dependence of phagocytic cup shape on membrane biophysical parameters

Using our model for the zipper mechanism, we have successfully simulated phagocytic engulfment in a broad range of parameter values (see Figure 2). Figure 2A shows two different characteristic cup shapes we obtained. Low surface tension (*i.e.* low energy cost for stretching the membrane and underlying actin cortex), and tight cell-volume constraint (*i.e.* high energy cost for increasing the cell volume), lead to a thin phagocytic cup since a thin cup requires extra membrane but little extra volume. In contrast, weak volume constraint and/or high surface tension produce a broad cup. Based on the parameters explored, we chose intermediate values for both surface tension and cell-volume constraint as our Standard Parameters (SP) for the remainder of our simulations in order to produce realistic cup shapes (see *Methods* for details). Figure 2B shows that most parameter values can be changed independently by at least one order of magnitude, without negatively affecting engulfment completion. Note that changing simultaneously several parameters may affect engulfment more drastically. Our simulations also show that cup shape depends on the kinetics of engulfment, determined by membrane fluctuations and therefore temperature (see Supplementrary Fig. S2). Additionally, preventing thermal fluctuations (by setting the temperature to zero Kelvin) during a simulation stops cup progression. This indicates that in our model membrane fluctuations are indeed required to bring receptors in close contact with ligand molecules on the particle, emphasizing their important role in the ratchet mechanism.

## Active versus passive uptake and the role of actin

Although phagocytosis generally involves active processes such as actin polymerization in the cup (active zipper) , recent reports indicate that phagocytosis may still work in an actin-independent manner. Indeed, phagocytic uptake was observed despite treating phagocytes with (modest amounts of) cytochalasin D . Hence, ligand-receptor binding may be sufficient in guiding the cell membrane around the particle under certain conditions (passive zipper). To investigate the energetic requirements of the zipper mechanism, we implemented simulations of the passive zipper. In these simulations, ligand-receptor bonds are not stabilized by actin polymerization and can unbind at later times, *i.e.* remain reversible. Hence, engulfment may still progress if the energetic cost of stretching and deforming the membrane is offset by the ligand-receptor binding energy in the presence of thermal membrane fluctuations.
Figure 3 (*left*) shows that engulfment of small ($1.5\mu$m radius) particles by the passive zipper leads to more variable phagocytic cup shapes than engulfment by the active zipper. For the active zipper, random membrane fluctuations are rectified by irreversible ligand-receptor binding due to actin polymerization. This leads to uniform progression of the cell membrane all around the particle at approximately the same speed (Figure 3A). In contrast, engulfment by the passive zipper occurs through binding of large membrane ruffles which eventually enclose the particle (Figure 3C). The variability of the phagocytic cup may be a measure of the respective contributions of active and passive processes in engulfment progression.
Figure 3 (*right*) compares time courses of the membrane energy and progression of uptake for the active and passive zippers. We found that in both cases, membrane energy decreases rapidly at the very beginning of the uptake process due to energetically favorable ligand-receptor binding without large, energetically unfavorable deformations of the cell membrane. After this short initial period, the total energy increases with simulation time. This increase is much more pronounced for the active zipper, which stabilizes energetically unfavorable random membrane fluctuations by actin polymerization. In contrast, for the passive zipper such high-energy membrane deformation may not last over time. The slower increase in energy for the passive zipper correlates with a slower engulfment. For the simulations shown, engulfment for the active zipper is approximately twice to three times as fast as for the passive zipper, although the latter eventually engulfs the particle. However, the difference between active and passive engulfment depends on biophysical parameters, and may be reduced for particular choices of the parameters values, *e.g.* lower surface tension and/or stronger ligand-receptor binding.

## Particle size matters for passive, not for active zipper

Experiments show that phagocytosis is relatively insensitive to particle size . Using our model for the active zipper, we simulated engulfment of spherical particles with different radii ranging from $1.2$ to $3.8\mu$m. Figure 2C shows that engulfment progresses normally for small and large particles. Hence the active zipper mechanism is sufficiently robust to allow engulfment of differently sized particles using the same set of biophysical parameters, although engulfment of large particles requires more time. Noticeably, large particles (in general, with radius larger than $2.5\mu$m) were taken up via more regular phagocytic cups than small particles (with radius $1.5\mu$m or smaller), indicating that active processes may be required for engulfment of large particles. To confirm this observation we have simulated engulfment of large $3\mu$m-radius particles by both the active and the passive zipper, shown in Supplementary Fig. S5. While the active zipper resulted in complete uptake of the particle, the passive zipper only engulfed a few percent of the particle's surface area. Thus the difference in phagocytic efficiency between the two zipper types was exacerbated for large particles, reflecting the importance of actin polymerization for engulfment of large particles.

## Experimental test of model predictions

To test our model predictions and to specifically compare active with passive engulfment for small and large particles, we transfected COS-7 cells with either wild-type Fc$\gamma$R (WT-Fc$\gamma$R) or a signaling-dead mutant receptor (Y282F/ Y298F-Fc$\gamma$R). Cells expressing the wild-type receptor are expected to perform active engulfment, whereas cells expressing the signaling-dead mutant receptor are expected to perform passive engulfment. As a control, passive engulfment is additionally implemented by treating cells expressing WT-Fc$\gamma$R with $0.2\mu$M of cytochalasin D (WT-Fc$\gamma$R$+$CytoD), which prevents actin polymerization (see *Methods*). Synchronized phagocytosis assays using small (1.5 $\mu$m radius) and large (3 $\mu$m radius) IgG-opsonized polystyrene particles were carried out, and, after fixation, receptor localization in phagocytic cups was visualized by fluorescence confocal microscopy. Cells were imaged at different time points during phagocytosis, and at each time point, three to eight imaged cells were each engulfing simultaneously between four and twenty particles (Figure 4A). Consequently, for each condition we analyzed at least seventy particles.

To test whether passive engulfment leads to more variable cups than active engulfment, we developed an image-analysis method illustrated in Figure 4B-F. The cup shape varibility was quantified by the standard deviation of the distribution of cell-membrane (Fc$\gamma$R-GFP fluorescence) height around the particle, divided by the square root of the average membrane height. The unit of membrane height is given by the distance ($0.4\mu$m) between consecutive confocal image planes (see *Methods*). Figure 5A shows that for small particles engulfed between 20 and 40% of their surfaces, cup variability increases from cells transfected with wild-type receptor to cells transfected with signaling-dead mutant receptor to WT-Fc$\gamma$+CytoD cells. The lowest variability, found for cells expressing wild-type receptor, is statistically significant against both passive zipper types (Student's t-test, p-value $<0.001$). This result is consistent with model predictions: Figure 5A, *inset* shows the cup variability from simulations, revealing that the active zipper leads to significantly less variable cups. In contrast, for the ranges of engulfment between 40 and 60% (Figure 5B) and between 60 and 100% (see Supplementary Fig. S7) we observed no noticeable difference in cup variability between the three experimental conditions, while our model consistently predicts more variable cups for the passive zipper (Figure 5B, *inset*). This discrepancy may indicate that active processes such as contraction by myosin motor proteins become important at later stages of engulfment, limiting our model's full validity to the early events in phagocytosis (see *Conclusion* section).
To confirm that our results are independent of the specifics of the analysis method used, we also analyzed phagocytic cups with an alternative, albeit less accurate, method (see Supplementary Fig. S9). This method consists of determining the distribution of Fc$\gamma$R-GFP fluorescence intensity around the particle at its equator plane, restricting the analysis to approximately half taken up particles ($30-70\%$). The standard deviation of this distribution provides an alternative measure of the cup variability. We arrived at the same conclusion, confirming our result of the difference in cup variability (see Supplementary Fig. 10). Finally, note that temperature-induced synchronization is imperfect and may lead to variability in cup growth . However, our measures of the variability in cup shape are independent of such an effect since we include all time points together in the analysis (except for plots showing the time dependence of engulfment).
Our experiments further show that cup shape ranges from regular to variable for all three experimental conditions, but that the frequency of different cup shapes depends on the condition. Figure 5C plots the repartition of phagocytic cups for different conditions into both regular and variable cups. Note that a cup was identified as regular if its variability was below the variability averaged over all experimental conditions. In contrast, a cup was identified as variable if its variability was above the overall average. This plot shows that in our experiments a regular cup is most likely produced by a cell expressing wild-type receptor, whereas a variable cup is most likely to be produced by a cytochalasin-D treated cell. Hence, the cup shape has universal features independent of biochemical details. Examples of a regular and a variable cup are provided in Figures 4E and F, respectively. Both cups were taken from a cytochalasin-D treated cell, confirming that a regular cup can occur under any of our experimental conditions.
Our model also predicts that uptake by the active zipper is significantly faster than with the passive zipper (see Figures 3B and D). We experimentally tested this prediction by determining the percentage of engulfed surface area for each particle for different time points after initiation of phagocytosis, and comparing this result with our simulations, in which simulation time was matched to actual time. Figure 6A shows that cells transfected with the wild-type receptor (active zipper) engulf significantly faster (three to four times) than cells under the other two conditions (passive zippers). This result is in quantitative accordance with our model predictions (Figure 6B). Furthermore, we determined the time dependence of phagocytic uptake for large particles. The active zipper, although slower for large than for small particles, still engulfs regularly, both in experiments (Figure 6C) and simulations (Figure 6D). Note that predicted and measured time courses are in very good agreement without rescaling the time axis of the large-particle simulation. Furthermore, Figures 6C and 6D demonstrate the inability of the passive zipper to take up large particles, in both experiments and simulations. After more than 10 minutes, the average engulfed surface area remains below 20%.

Note that for the time points measured, the average uptake does not exceed 50%, even for WT Fc$\gamma$R. This is caused by the fact that some particles are not engulfed irrespective of the condition (see Supplementary Fig. S6), reducing the average percentage of engulfment. The proportion of almost completely engulfed particles (with engulfed surface area larger than 70%) beyond 6 minutes is represented in the inset of Figure 6A, showing that completion of engulfment is possible even for cells without actin polymerization. Note that long phagocytic assays (12, 14, and 45 minutes) were performed for cytochalasin-D treated cells only, explaining why the difference in complete engulfment with cells expressing the wild-type receptor is smaller than one may expect.
From these results, we conclude that functional actin polymerization is required for fast and regular engulfment in phagocytosis. Nevertheless, in line with our model predictions, cells showing deficient actin polymerization at cups are still able to uptake small particles, although more slowly and with more variable cups.

## Active zipper reproduces particle-shape dependence of phagocytosis

Previously published experiments show that phagocytosis depends strongly on particle shape. In particular, elongated particles (similar to rod-shaped bacteria) are only taken up when presented to the phagocyte with their tip first . To test the particle-shape dependence of phagocytosis we conducted simulations of the active zipper using particles of different shapes while varying the initial orientation of the particle on the cell surface. Figure 2D shows the uptake of a prolate spheroid for two different orientations on the cell membrane after the same elapsed simulation time. In accordance with experimental observations , uptake is more advanced for the spheroid particle engulfed with its tip first (about 80% engulfed surface area) than for the particle attached along its major axis (about 50% engulfed in the same amount of time). This suggests a strong inhibitory effect of high local curvature on uptake. In our model this is readily attributed to the energetic cost of bending the membrane around the two highly curved ends of an elongated particle placed horizontally on the cell membrane. In line with this explanation and experiments , the spiral-shaped particle in Figure 2E is not engulfed after a time duration sufficient for the engulfment of large spherical particle of twice its volume. Hence, our simulations demonstrate that particle shape and orientation are indeed important biophysical parameters for phagocytosis.

# Conclusion

In this work, we studied the biophysical requirements of the zipper mechanism, in particular the role of receptor-induced actin polymerization, and the effect of particle shape on uptake. In our model, the underlying biophysical mechanism of the zipper is an actin-driven thermal ratchet, which renders random membrane fluctuations irreversible close to the particle (Figure 1). This mechanism is supported by several recent experiments . Previously, such Brownian ratchet models were successfully applied to explain force generation by actin polymerization and motility of the pathogen *Listeria* in hosts cells . Our fully stochastic simulations can address for the first time the variability in particle uptake, recently noticed in phagocytic cup growth and completion of endocytosis . Implementation of our model in simulations led indeed to phagocytic engulfment for a broad range of values of membrane parameters (Figure 2), indicating exquisit robustness of the phagocytic process. However, phagocytic cup shape depends on parameter values, specifically on the ratio between surface tension and cell-volume constraint, as well as on the kinetics of engulfment. Cells with low surface tension and/or tight volume constraint develop thin cups (Figure 2A, *left*), characteristic of Fc$\gamma$R-mediated phagocytosis . In contrast, cells with high surface tension produce broad cups (Figure 2A, *right*). The latter cup shape is more reminiscent of CR3-mediated phagocytosis, although for this type of phagocytosis particles are believed to sink into cells without protrusive cups . Using our model we were able to address the question whether the zipper mechanism requires an active driving force, such as provided by actin polymerization. For this purpose, we compared the regular active zipper with a passive version of the zipper. In the passive zipper, ligand-receptor bonds are as strong as for the active zipper (based on experimental observation ) but are reversible, *i.e.* are not supported by actin polymerization. We demonstrated that the passive zipper also leads to engulfment of small particles (of radius $1.5\mu$m), although cup progression is slower and more variable (see Figure 3). Our active zipper can also reproduce the independence of uptake on particle size, in line with experimental observations . In contrast, large particles (of radius $3\mu$m) are poorly phagocytozed by the passive zipper. We subsequently confirmed these predictions with experiments by transfecting COS-7 fibroblasts with wild-type Fc$\gamma$R and signaling-dead mutant Y282F/Y298F Fc$\gamma$R (see Figures 5 and 6). While the wild-type receptor represents the active zipper, the passive zipper is implemented through the use of signaling-dead mutant receptor or treatment with cytochalasin D. Both prevent actin polymerization in the cups. Our study may indicate that ancient forms of phagocytosis were driven by physical (passive) principles, and only later in evolution biochemical regulatory pathways were added for further support and robustness. Passive phagocytosis may also become important when energy sources are scarce. Despite the robustness of phagocytosis to particle size, there appears to be a mechanical bottleneck around half-engulfment, recently observed by imaging and also predicted by our model. For slight variations in some of the parameters, our simulations of the active zipper produce either complete or significantly incomplete uptake (see Supplementary Fig. S6). Indeed, when the cup grows, deforming the membrane costs more and more energy per surface area engulfed. Beyond half-engulfment, the surface tension energy is twice as high as at the beginning of engulfment due to the membrane folding back onto itself. If this energy cannot be provided by the zipper, then cup progression stalls and the particle remains incompletely engulfed. Alternatively, the experimental data on incomplete particle uptake may be the result of particle attachment on cell-membrane areas, unable to phagocytose due to other reasons, such as unfavorable local cell-surface curvature, proximity to cell edge or nucleus, or missing proteins, lipids, and smaller molecules belonging to key signaling pathways. Further studies will be required, ideally using live-cell imaging to avoid the need for conserving the cell's internal structures by fixation .
Our model for the zipper mechanism can also explain the strong particle-shape dependence observed in phagocytosis. Experiments show that an elongated spheroid is rapidly engulfed if the particle attaches to the cell membrane with its tip, but not if the particle attaches along its major axis . Furthermore, spiral-shaped particles are not phagocytozed . The strong particle-shape dependence of phagocytosis is likely of biological relevance. On the one hand, it may increase the rate of infection of host cells by pathogenic bacteria. Indeed, recent experiments show that *Mycobacteria tuberculosis* and *marinum* are efficiently taken up and later released for spreading of the infection with the bacteria's tip first . On the other hand, the highly curved shapes of some bacteria, *e.g.* the spiral-shaped *Helicobacter* and *Campylobacter* species, may prevent their uptake by macrophages , although injection of effector proteins can also be used by pathogens to hijack the immune or host cell's phagocytic response . Furthermore, the particle-shape dependence of phagocytosis may be exploited to improve drug delivery by enclosing active drugs in particles, whose shape prevents uptake and destruction by macrophages .
While our biophysical model for the zipper mechanism is readily accessible for analysis and interpretation, the small number of model parameters makes a direct comparison with measured parameter values difficult. First, our membrane parameters such as surface tension and bending stiffness are an order of magnitude smaller than reported bulk membrane parameters (see *Methods*). This reduction is not surprising as cells regulate these parameters locally for efficient uptake . Such regulation may include lowering of the surface tension by local membrane delivery through vesicles and unfolding of membrane wrinkles , as well as changes in the lipid and protein composition in the phagocytic cup . Second, our description of ligand-receptor interaction assumes that ligand and receptor distributions are continuous and homogeneous, while experiments indicate the formation of receptor micro-clusters , possibly as part of lipid rafts . Third, our mechanism for actin polymerization based on membrane fluctuations neglects the role of the motor proteins (myosin I and II), whose role in membrane deformations has been established . Consequently, our simulations describe well the dynamics of engulfment during the first two thirds of the uptake. At later stages of uptake, experiments show that phagocytic cups close rapidly with a thin membrane protrusion , while our simulations show slow cup closure. However, taking into account myosin-driven contraction is beyond the scope of the current work.
Our model may also be applicable to other biological systems in which a zipper-like mechanism is involved. One such example is sporulation of *Bacillus subtilis* during starvation. After asymmetric cell division, the larger mother cell engulfs the smaller forespore for spore maturation. Interestingly, the mother cell even engulfs the forespore when the cell wall is artificially removed. This process occurs in a fast, zipper-like fashion without known sources of energy . Importantly, forespore engulfment is subject to high variation. About 60% of the cells successfully complete forespore engulfment, while 40% do not at all, similar to the observation of the mechanical bottleneck in phagocytosis. Other examples of engulfment may not be driven by a zipper mechanism. For instance, the penetration of red blood cells by the malaria *Plasmodium* merozoite is driven by an elaborate actin machinery of the particle (the parasite), which devotes all its resources to wrap the cell membrane around itself. In stark contrast to phagocytosis, the engulfing host cell is completely passive .

# Methods

## Theoretical techniques

The cell membrane is described by a two dimensional elastic sheet , which includes both its lipid bilayer and associated actin cortex. The particle is assumed to be rigid and immobile. Moreover, we account for ligand-receptor binding and include a constraint on the cell volume. Therefore, the total free energy is given by $$E=E_{\text{m}}+E_{\text{vol}}+E_{\text{LR}} \label{eq:free_energy_total} \text{,}$$ where $$E_{\text{m}}=\int_{m} d^{2}\mathbf{r}\left[\frac{\kappa_{b}}{2} C^2(\mathbf{r})+\sigma\right] \label{eq:free_energy_membrane}$$ with $C^2(\mathbf{r})=C{_1}^{2}(\mathbf{r})+C{_2}^{2}(\mathbf{r})$ the square curvature obtained from the minimal ($C{_1}$) and maximal ($C{_2}$) curvatures at point $\mathbf{r}$. Note that the term corresponding to the product $C{_1}(\mathbf{r})C{_2}(\mathbf{r})$ is independent of the actual shape of the membrane as long as the overall topology is conserved, and therefore is ignored (Gauss-Bonnet theorem). Bending stiffness $\kappa_{b}$ reflects the energy cost of bending, and surface tension $\sigma$ reflects the energy cost of stretching the membrane with underlying actin cortex. Furthermore, expanding or shrinking the cytosol locally costs the energy $$E_{\text{vol}}=E_{\text{cell}}(V)-E_{\text{cell}}(V_0)=\kappa_{P}(V-V_0)^2 \label{eq:free_energy_volume} \text{,}$$ where the quadratic dependence on actual volume $V$ comes from the lowest order Taylor expansion of the cell energy around local steady state volume $V_0$, and $\kappa_{P}=\frac{1}{2}\left( \frac{\partial^2 E_{cell}}{\partial V^2 }\right)_{V=V_0}$. Taylor expansion is justified by the fact that our simulations and experiments use particles significantly smaller than the cell (at least 10-20 times in volume). Finally, in our model ligand-receptor binding is not described explicitly at the molecular scale, but is accounted for by a membrane-particle contact potential $V_{LR}(\mathbf{r})$, where $V_{LR}(\mathbf{r})=-V_{LR}^{0}$ if a membrane patch is within a distance $R_0$ of the particle and zero if further away. Specifically, the associated energy is given by $$E_{\text{LR}}=\int_{m} d^{2}\mathbf{r}\text{ }V_{\text{LR}}(\mathbf{r}) \label{eq:free_energy_LR}\text{,}$$ where the integral is performed over the cell-membrane area. Effectively, $V_{LR}^{0}$ is given by the product of the individual ligand-receptor binding energy and the density of ligand-receptor bonds, divided by the density of vertices on the model membrane (see below). The width of the square potential $R_0$ is chosen to be very small compared to the other length-scales of the model $R_0<0.1R$ and does not influence the results.

Simulations of phagocytic engulfment were implemented by discretizing the cell and particle surfaces using the Surface Evolver software . This software is designed to perform energy minimization on flexible surfaces, and is freely available from http://www.susqu.edu/facstaff/ b/brakke/evolver. The software includes a built-in programming language, which we used to implement a Monte Carlo algorithm (see below). The cell membrane is approximated by a finite number of vertices, used to create a triangular mesh. The software computes the local energy density at each vertex and sums up the energy contributions from all the surface elements to obtain the total free energy Eq. 1.

Our model uses four tunable biophysical parameters. Unless otherwise specified, we have used the set of Standard Parameters (SP), chosen according to experimental measurements when possible (see Supplementary Information, section 1), but ultimately to produce realistic cup shapes (see Figure 2). This set of parameters includes: the cell membrane bending rigidity $\kappa_b$ and surface tension $\sigma$, respectively set to $1.3 \times 10^{-2}\text{ pN}\text{}\mu \text{m}$ and $6.2 \times 10^{-6}\text{ mNm}^{-1}$, *i.e.* slightly below the experimentally measured values since local changes in chemical composition of the cups membrane may reduce these parameters . The third parameter is the total binding energy density $\epsilon=58.5\text{ pN}\text{}\mu \text{m}^{-1}$. This value was estimated from measurements of the individual Fc$\gamma$R-IgG binding free energy $\Delta F_{LR} \approx 20k_{B}T$ , the average density $d_{LR}=270-435\text{ }\mu$m$^{-2}$ of IgG-Fc$\gamma$R bonds , and the fact that in response to diffusion and trapping or signaling, receptors may cluster at the cup. Finally, the local constraint on cell-volume has been chosen $\kappa_{P}=2.56 \times 10^{-5}\text{ pN}\text{}\mu \text{m}^{-5}$ to allow 20 percent volume variation in line with observation .

The Surface Evolver was only used to obtain a triangular mesh (vertices connected by edges) of the cell membrane, and to resample the membrane as the uptake progresses. The cell-membrane evolution was implemented using finite-temperature Monte Carlo Metropolis simulations . Details of the simulation can be found in the Supplementary Information, section 2. Briefly, our algorithm calculates the total energy of the initial membrane configuration, then randomly selects a point to be the center of a membrane fluctuation, and a random direction and lateral extension of the fluctuation. Subsequently, the energy of the new membrane configuration is calculated, and compared to the initial energy. If the membrane fluctuation decreased the energy, the trial fluctuation is accepted and the procedure is reiterated starting from the new membrane configuration. To the contrary, if the membrane energy increased with the trial fluctuation, the latter may be rejected with some probability depending on the configuration's energy difference. In this case, a new fluctuation is attempted from the initial configuration. Between trial fluctuations, the cell-membrane vertices are examined. For the active zipper, the vertices within the closed neighborhood of the particle are immobilized for the remainder of the simulation. For the passive zipper, every membrane fluctuation may be reversed at a later time.

(1) The amplitude of a membrane fluctuation. This parameter is set to $0.5R_0$ in all the simulations. (2) The mesh size, describing the maximal distance between two neighboring vertices. This parameter is set to $R_0$. (3) Minimal width of a membrane fluctuation. To ensure that a minimum number of vertices is involved in each fluctuation, this parameter is set to $4R_0$. (4) Mesh refinement range. Our simulation script automatically refines the mesh locally around a previously immobilized vertex within this range. This parameter is set to the particle radius $R$, leading to reasonably smooth cup shapes in a reasonably short calculation time (1-2 days for complete uptake using a Intel(R) Core(TM)2 Quad CPU working at 2.50GHz and run by the RedHat EL5 linux distribution with 4GB RAM).

## Experimental techniques

COS-7 fibroblast cells were obtained from American Type Culture Collection (ATCC) and cultured in Dulbecco's Modified Eagle's Medium (DMEM), supplemented with 10% foetal bovine serum (FBS) and penicillin/streptomycin (Invitrogen). cDNAs encoding wild-type and Y282F/Y298F human Fc$\gamma$RIIa from pRK5-Fc$\gamma$RIIa and pRK5-Y282F/Y298F-Fc$\gamma$RIIa were subcloned into pEGFP-N1 (Clontech) using primers 5'-ggtccaactgcacctcggt-3' and 5'-ccccccgaattctgttattactgttgacatggtc-3'. The cytoplasmic tail truncation mutant 239-Fc$\gamma$RIIa was generated from the pRK5-Fc$\gamma$RIIa template using primers 5'-ggtccaactgcacctcggt-3' and 5'-gggggggaattctcctgcagtagatcaaggccact-3'. Rabbit anti-bovine serum albumin (BSA) serum was purchased from Sigma-Aldrich. Alexa-conjugated secondary antibodies and phalloidin were purchased from Invitrogen.

COS-7 cells were transfected with GFP-tagged Fc$\gamma$RIIa constructs using an Amaxa Nucleofector and Nucleofector cell line kit R following the manufacturer instructions. For phagocytosis assays, transfected cells were seeded onto glass coverslips in 24-well plates at a density of 15,000 cells/coverslip and incubated at 37$\,^{\circ}$C for 72 h. 1 hour before commencement of phagocytosis assays, cells were incubated for 1 hour at 37$\,^{\circ}$C with serum-free DMEM plus 10 mM Hepes (Invitrogen). $1.5\mu$m- and $3\mu$m-radius latex-polystyrene particles (Sigma-Aldrich) were opsonized by first incubating overnight at 4$\,^{\circ}$C with 3% BSA fraction V in PBS (Sigma-Aldrich) followed by incubation with 1:100 dilution of rabbit anti-BSA in PBS for 1 hour at room temperature. Particles were re-suspended in ice-cold serum-free DMEM plus 10 mM Hepes at a concentration of 1.5x10$^6$ particles/ml and 500 $\mu$l added to each coverslip. Plates were incubated on ice for 10 minutes to allow binding of particles. Medium was then replaced with pre-warmed serum-free DMEM plus 10 mM Hepes and plates were incubated at 37$\,^{\circ}$C, then processed for scoring or microscopy as described below. Experiments carried out on cells treated with cytochalasin D were conducted as above with the addition of a 20 min pre-incubation step with 0.2 $\mu$M cytochalasin D in serum-free DMEM plus 10 mM Hepes at 37$\,^{\circ}$C immediately before incubation of cells with opsonized particles. This concentration of cytochalasin D was included in all further incubation steps.

Plates were placed on ice after a 20 minute incubation at 37$\,^{\circ}$C and medium replaced with a 1:500 dilution of anti-rabbit Alexa 488 in 3% BSA/PBS at 4$\,^{\circ}$C for 5 min. Cells were fixed after incubation at 37$\,^{\circ}$C for the appropriate amount of time with ice-cold 4% paraformaldehyde/PBS, permeabilised and labelled with goat anti-rabbit Alexa 633, and phalloidin Alexa 555 for visualizing F-actin at room temperature for 30 min. Z-series image stacks were acquired on a Zeiss LSM-510 confocal microscope using a step size of 0.4 $\mu$m.

Two fluorescence channels (IgG, Fc$\gamma$R-GFP) were acquired and analyzed using MATLAB (MathWorks). During the acquisition process, the Fc$\gamma$R-GFP fluorescence intensity was set to zero at any pixel where the IgG intensity is null. The fluorescence intensity distribution of IgG was used to determine the coordinates of the centers of particles with their corresponding radii, using an automated search based on the Hough transform , available online at www.mathworks.com/matlabcentral/fileexchange/4985 within each 2D image. The percentage of engulfed particle surface area was calculated by comparing the local Fc$\gamma$R-GFP (cell membrane) and IgG (particle surface) intensity distributions within a sphere $S_0$ of radius $4R/3$ whose center coincides with the particle center. Two methods were used to quantify the variability of the cups. The method used in the main text cuts the three-dimensional image (and hence the circular projections of particles within each imaging plane) in twenty-four angular segments, and finds the highest plane in which Fc$\gamma$R-GFP fluorescence intensity is detected in the immediate neighborhood of the particle for each segment (see Figure 4). This analysis produces a distribution of membrane height *versus* angular segment index, for which we compute the average and the mean-square deviation. The average height reached by cell membrane is roughly proportional to the surface engulfed (see Supplementary Fig. S10), and the mean-square deviation divided by the square root of the average height quantifies the variability of the phagocytic cup, excluding a trivial size dependence on the variability. The less accurate alternative method determines the angular distribution of Fc$\gamma$R-GFP fluorescence intensity within the particle's equator plane in the particle's immediate neighborhood, keeping only particles whose uptake level comprises between $30$ and $70\%$ (roughly half-engulfed particles). Specifically, this region is cut into twenty-four identical angular segments, for which the total Fc$\gamma$R fluorescence intensity is calculated. Then the average intensity per segment is calculated, as well as the standard deviation. The higher the standard deviation, the more variable the cup.

# Accession Numbers

The Fc$\gamma$RIIa receptor is referenced in protein database Genbank (http://www.ncbi.nlm.nih.gov/Genbank) under the accession number CAA01563. Signaling-dead mutant receptor Y282F/Y298F-Fc$\gamma$RIIa is obtained by replacing tyrosines (Y) with phenylalanines (F) at positions 282 and 298.

# Authors contributions

ST participated to the conception of the model, and designed the simulation and image analysis algorithms. He carried out the simulations, image analysis, and statistical analysis of data as well as drafted the manuscript. AD and GT carried out the fluorescence imaging experiments and participated in writing the manuscript. RGE participated to the conception of the model and the interpretation of both theoretical and experimental results, and contributed to the writing of the manuscript. All authors read and approved the final manuscript.

# Acknowledgements

This paper is dedicated to the memory of Emmanuelle Caron, who tragically passed away in 2009. We acknowledge Micah Dembo and Günther Gerisch for helpful discussions, and Ken Brakke for help with the Surface Evolver software. We thank Vania Braga, Tony Magee and Brian Robertson for careful reading of the manuscript, and Suhail Islam for computational support. All authors would like to acknowledge funding from the Center for Integrative Systems Biology at Imperial College (CISBIC). RGE was additionally supported by the Biotechnology and Biological Sciences Research Council grant BB/G000131/1.


author: G. E. Hinton$^\ast$, N. Srivastava, A. Krizhevsky, I. Sutskever and R. R. Salakhutdinov
Department of Computer Science, University of Toronto,
6 King's College Rd, Toronto, Ontario M5S 3G4, Canada

$^\ast$To whom correspondence should be addressed; E-mail: email@example.com
bibliography: dropout.bib
title: Improving neural networks by preventing co-adaptation of feature detectors

> **When a large feedforward neural network is trained on a small training set, it typically performs poorly on held-out test data. This "overfitting" is greatly reduced by randomly omitting half of the feature detectors on each training case. This prevents complex co-adaptations in which a feature detector is only helpful in the context of several other specific feature detectors. Instead, each neuron learns to detect a feature that is generally helpful for producing the correct answer given the combinatorially large variety of internal contexts in which it must operate. Random "dropout" gives big improvements on many benchmark tasks and sets new records for speech and object recognition.**

A feedforward, artificial neural network uses layers of non-linear "hidden" units between its inputs and its outputs. By adapting the weights on the incoming connections of these hidden units it learns feature detectors that enable it to predict the correct output when given an input vector . If the relationship between the input and the correct output is complicated and the network has enough hidden units to model it accurately, there will typically be many different settings of the weights that can model the training set almost perfectly, especially if there is only a limited amount of labeled training data. Each of these weight vectors will make different predictions on held-out test data and almost all of them will do worse on the test data than on the training data because the feature detectors have been tuned to work well together on the training data but not on the test data.

Overfitting can be reduced by using "dropout" to prevent complex co-adaptations on the training data. On each presentation of each training case, each hidden unit is randomly omitted from the network with a probability of 0.5, so a hidden unit cannot rely on other hidden units being present. Another way to view the dropout procedure is as a very efficient way of performing model averaging with neural networks. A good way to reduce the error on the test set is to average the predictions produced by a very large number of different networks. The standard way to do this is to train many separate networks and then to apply each of these networks to the test data, but this is computationally expensive during both training and testing. Random dropout makes it possible to train a huge number of different networks in a reasonable time. There is almost certainly a different network for each presentation of each training case but all of these networks share the same weights for the hidden units that are present.

We use the standard, stochastic gradient descent procedure for training the dropout neural networks on mini-batches of training cases, but we modify the penalty term that is normally used to prevent the weights from growing too large. Instead of penalizing the squared length (L2 norm) of the whole weight vector, we set an upper bound on the L2 norm of the incoming weight vector for each individual hidden unit. If a weight-update violates this constraint, we renormalize the weights of the hidden unit by division. Using a constraint rather than a penalty prevents weights from growing very large no matter how large the proposed weight-update is. This makes it possible to start with a very large learning rate which decays during learning, thus allowing a far more thorough search of the weight-space than methods that start with small weights and use a small learning rate.

At test time, we use the "mean network" that contains all of the hidden units but with their outgoing weights halved to compensate for the fact that twice as many of them are active. In practice, this gives very similar performance to averaging over a large number of dropout networks. In networks with a single hidden layer of $N$ units and a "softmax" output layer for computing the probabilities of the class labels, using the mean network is exactly equivalent to taking the geometric mean of the probability distributions over labels predicted by all $2^N$ possible networks. Assuming the dropout networks do not all make identical predictions, the prediction of the mean network is guaranteed to assign a higher log probability to the correct answer than the mean of the log probabilities assigned by the individual dropout networks . Similarly, for regression with linear output units, the squared error of the mean network is always better than the average of the squared errors of the dropout networks.

We initially explored the effectiveness of dropout using MNIST, a widely used benchmark for machine learning algorithms. It contains 60,000 28x28 training images of individual hand written digits and 10,000 test images. Performance on the test set can be greatly improved by enhancing the training data with transformed images or by wiring knowledge about spatial transformations into a convolutional neural network or by using generative pre-training to extract useful features from the training images without using the labels . Without using any of these tricks, the best published result for a standard feedforward neural network is 160 errors on the test set. This can be reduced to about 130 errors by using 50% dropout with separate L2 constraints on the incoming weights of each hidden unit and further reduced to about 110 errors by also dropping out a random 20% of the pixels (see figure ).

Dropout can also be combined with generative pre-training, but in this case we use a small learning rate and no weight constraints to avoid losing the feature detectors discovered by the pre-training. The publically available, pre-trained deep belief net described in got 118 errors when it was fine-tuned using standard back-propagation and 92 errors when fine-tuned using 50% dropout of the hidden units. When the publically available code at URL was used to pre-train a deep Boltzmann machine five times, the unrolled network got 103, 97, 94, 93 and 88 errors when fine-tuned using standard backpropagation and 83, 79, 78, 78 and 77 errors when using 50% dropout of the hidden units. The mean of 79 errors is a record for methods that do not use prior knowledge or enhanced training sets (For details see Appendix ).

We then applied dropout to TIMIT, a widely used benchmark for recognition of clean speech with a small vocabulary. Speech recognition systems use hidden Markov models (HMMs) to deal with temporal variability and they need an acoustic model that determines how well a frame of coefficients extracted from the acoustic input fits each possible state of each hidden Markov model. Recently, deep, pre-trained, feedforward neural networks that map a short sequence of frames into a probability distribution over HMM states have been shown to outperform tradional Gaussian mixture models on both TIMIT and a variety of more realistic large vocabulary tasks .

Figure  shows the frame *classification* error rate on the core test set of the TIMIT benchmark when the central frame of a window is classified as belonging to the HMM state that is given the highest probability by the neural net. The input to the net is 21 adjacent frames with an advance of 10ms per frame. The neural net has 4 fully-connected hidden layers of 4000 units per layer and 185 "softmax" output units that are subsequently merged into the 39 distinct classes used for the benchmark. Dropout of 50% of the hidden units significantly improves classification for a variety of different network architectures (see figure ). To get the frame *recognition* rate, the class probabilities that the neural network outputs for each frame are given to a decoder which knows about transition probabilities between HMM states and runs the Viterbi algorithm to infer the single best sequence of HMM states. Without dropout, the recognition rate is $22.7$% and with dropout this improves to $19.7$%, which is a record for methods that do not use any information about speaker identity.

CIFAR-10 is a benchmark task for object recognition. It uses 32x32 downsampled color images of 10 different object classes that were found by searching the web for the names of the class (e.g. dog) or its subclasses (e.g. Golden Retriever). These images were labeled by hand to produce 50,000 training images and 10,000 test images in which there is a single dominant object that could plausibly be given the class name (see figure ). The best published error rate on the test set, without using transformed data, is 18.5% . We achieved an error rate of 16.6% by using a neural network with three convolutional hidden layers interleaved with three "max-pooling" layers that report the maximum activity in local pools of convolutional units. These six layers were followed by one locally-connected layer (For details see Appendix ) . Using dropout in the last hidden layer gives an error rate of 15.6%.

ImageNet is an extremely challenging object recognition dataset consisting of thousands of high-resolution images of thousands of classes of object . In 2010, a subset of 1000 classes with roughly 1000 examples per class was the basis of an object recognition competition in which the winning entry, which was actually an average of six separate models, achieved an error rate of 47.2% on the test set. The current state-of-the-art result on this dataset is 45.7% . We achieved comparable performance of 48.6% error using a single neural network with five convolutional hidden layers interleaved with "max-pooling" layer followed by two globally connected layers and a final 1000-way softmax layer. All layers had L2 weight constraints on the incoming weights of each hidden unit. Using 50% dropout in the sixth hidden layer reduces this to a record 42.4% (For details see Appendix ).

For the speech recognition dataset and both of the object recognition datasets it is necessary to make a large number of decisions in designing the architecture of the net. We made these decisions by holding out a separate validation set that was used to evaluate the performance of a large number of different architectures and we then used the architecture that performed best with dropout on the validation set to assess the performance of dropout on the real test set.

The Reuters dataset contains documents that have been labeled with a hierarchy of classes. We created training and test sets each containing 201,369 documents from 50 mutually exclusive classes. Each document was represented by a vector of counts for 2000 common non-stop words, with each count $C$ being transformed to $\log(1+C)$. A feedforward neural network with 2 fully connected layers of 2000 hidden units trained with backpropagation gets 31.05% error on the test set. This is reduced to 29.62% by using 50% dropout (Appendix ).

We have tried various dropout probabilities and almost all of them improve the generalization performance of the network. For fully connected layers, dropout in all hidden layers works better than dropout in only one hidden layer and more extreme probabilities tend to be worse, which is why we have used 0.5 throughout this paper. For the inputs, dropout can also help, though it it often better to retain more than 50% of the inputs. It is also possible to adapt the individual dropout probability of each hidden or input unit by comparing the average performance on a validation set with the average performance when the unit is present. This makes the method work slightly better. For datasets in which the required input-output mapping has a number of fairly different regimes, performance can probably be further improved by making the dropout probabilities be a learned function of the input, thus creating a statistically efficient "mixture of experts" in which there are combinatorially many experts, but each parameter gets adapted on a large fraction of the training data.

Dropout is considerably simpler to implement than Bayesian model averaging which weights each model by its posterior probability given the training data. For complicated model classes, like feedforward neural networks, Bayesian methods typically use a Markov chain Monte Carlo method to sample models from the posterior distribution . By contrast, dropout with a probability of $0.5$ assumes that all the models will eventually be given equal importance in the combination but the learning of the shared weights takes this into account. At test time, the fact that the dropout decisions are independent for each unit makes it very easy to approximate the combined opinions of exponentially many dropout nets by using a single pass through the mean net. This is far more efficient than averaging the predictions of many separate models.

A popular alternative to Bayesian model averaging is "bagging" in which different models are trained on different random selections of cases from the training set and all models are given equal weight in the combination . Bagging is most often used with models such as decision trees because these are very quick to fit to data and very quick at test time. Dropout allows a similar approach to be applied to feedforward neural networks which are much more powerful models. Dropout can be seen as an extreme form of bagging in which each model is trained on a single case and each parameter of the model is very strongly regularized by sharing it with the corresponding parameter in all the other models. This is a much better regularizer than the standard method of shrinking parameters towards zero.

A familiar and extreme case of dropout is "naive bayes" in which each input feature is trained separately to predict the class label and then the predictive distributions of all the features are multiplied together at test time. When there is very little training data, this often works much better than logistic classification which trains each input feature to work well in the context of all the other features.

Finally, there is an intriguing similarity between dropout and a recent theory of the role of sex in evolution . One possible interpretation of the theory of mixability articulated in  is that sex breaks up sets of co-adapted genes and this means that achieving a function by using a large set of co-adapted genes is not nearly as robust as achieving the same function, perhaps less than optimally, in multiple alternative ways, each of which only uses a small number of co-adapted genes. This allows evolution to avoid dead-ends in which improvements in fitness require co- ordinated changes to a large number of co-adapted genes. It also reduces the probability that small changes in the environment will cause large decreases in fitness – a phenomenon which is known as "overfitting" in the field of machine learning.

# Experiments on MNIST

## Details for dropout training

The MNIST dataset consists of 28 $\times$ 28 digit images - 60,000 for training and 10,000 for testing. The objective is to classify the digit images into their correct digit class. We experimented with neural nets of different architectures (different number of hidden units and layers) to evaluate the sensitivity of the dropout method to these choices. We show results for 4 nets (784-800-800-10, 784-1200-1200-10, 784-2000-2000-10, 784-1200-1200-1200-10). For each of these architectures we use the same dropout rates - 50% dropout for all hidden units and 20% dropout for visible units. We use stochastic gradient descent with 100-sized minibatches and a cross-entropy objective function. An exponentially decaying learning rate is used that starts at the value of 10.0 (applied to the average gradient in each minibatch). The learning rate is multiplied by 0.998 after each epoch of training. The incoming weight vector corresponding to each hidden unit is constrained to have a maximum squared length of $l$. If, as a result of an update, the squared length exceeds $l$, the vector is scaled down so as to make it have a squared length of $l$. Using cross validation we found that $l = 15$ gave best results. Weights are initialzed to small random values drawn from a zero-mean normal distribution with standard deviation 0.01. Momentum is used to speed up learning. The momentum starts off at a value of 0.5 and is increased linearly to 0.99 over the first 500 epochs, after which it stays at 0.99. Also, the learning rate is multiplied by a factor of (1-momentum). No weight decay is used. Weights were updated at the end of each minibatch. Training was done for 3000 epochs. The weight update takes the following form: $$\begin{aligned}
\Delta w^{t} & = & p^{t}\Delta w^{t-1} - (1-p^{t})\epsilon^{t}\langle \nabla_w L \rangle \\
w^{t} & = & w^{t-1} + \Delta w^{t},
\end{aligned}$$ where, $$\begin{aligned}
\epsilon^{t} & = & \epsilon_{0}f^{t}\\
p^{t} &=& \begin{cases}
            \frac{t}{T}p_{i} + (1-\frac{t}{T})p_{f} & t < T \\
            p_{f} & t \geq T
        \end{cases}
\end{aligned}$$ with $\epsilon_0 = 10.0$, $f = 0.998$, $p_{i} = 0.5$, $p_{f} = 0.99$, $T = 500$. While using a constant learning rate also gives improvements over standard backpropagation, starting with a high learning rate and decaying it provided a significant boost in performance. Constraining input vectors to have a fixed length prevents weights from increasing arbitrarily in magnitude irrespective of the learning rate. This gives the network a lot of opportunity to search for a good configuration in the weight space. As the learning rate decays, the algorithm is able to take smaller steps and finds the right step size at which it can make learning progress. Using a high final momentum distributes gradient information over a large number of updates making learning stable in this scenario where each gradient computation is for a different stochastic network.

## Details for dropout finetuning

Apart from training a neural network starting from random weights, dropout can also be used to finetune pretrained models. We found that finetuning a model using dropout with a small learning rate can give much better performace than standard backpropagation finetuning.

*Deep Belief Nets* - We took a neural network pretrained using a Deep Belief Network . It had a 784-500-500-2000 architecture and was trained using greedy layer-wise Contrastive Divergence learning [^1]. Instead of fine-tuning it with the usual backpropagation algorithm, we used the dropout version of it. Dropout rate was same as before : 50% for hidden units and 20% for visible units. A constant small learning rate of 1.0 was used. No constraint was imposed on the length of incoming weight vectors. No weight decay was used. All other hyper-parameters were set to be the same as before. The model was trained for 1000 epochs with stochstic gradient descent using minibatches of size 100. While standard back propagation gave about 118 errors, dropout decreased the errors to about 92.

*Deep Boltzmann Machines* - We also took a pretrained Deep Boltzmann Machine [^2] (784-500-1000-10) and finetuned it using dropout-backpropagation. The model uses a 1784 - 500 - 1000 - 10 architecture (The extra 1000 input units come from the mean-field activations of the second layer of hidden units in the DBM, See  for details). All finetuning hyper-parameters were set to be the same as the ones used for a Deep Belief Network. We were able to get a mean of about 79 errors with dropout whereas usual finetuning gives about 94 errors.

## Effect on features

One reason why dropout gives major improvements over backpropagation is that it encourages each individual hidden unit to learn a useful feature without relying on specific other hidden units to correct its mistakes. In order to verify this and better understand the effect of dropout on feature learning, we look at the first level of features learned by a 784-500-500 neural network without any generative pre-training. The features are shown in Figure . Each panel shows 100 random features learned by each network. The features that dropout learns are simpler and look like strokes, whereas the ones learned by standard backpropagation are difficult to interpret. This confirms that dropout indeed forces the discriminative model to learn good features which are less co-adapted and leads to better generalization.

# Experiments on TIMIT

The TIMIT Acoustic-Phonetic Continuous Speech Corpus is a standard dataset used for evaluation of automatic speech recognition systems. It consists of recordings of 630 speakers of 8 dialects of American English each reading 10 phonetically-rich sentences. It also comes with the word and phone-level transcriptions of the speech. The objective is to convert a given speech signal into a transcription sequence of phones. This data needs to be pre-processed to extract input features and output targets. We used Kaldi, an open source code library for speech [^3], to pre-process the dataset so that our results can be reproduced exactly. The inputs to our networks are log filter bank responses. They are extracted for 25 ms speech windows with strides of 10 ms.

Each dimension of the input representation was normalized to have mean 0 and variance 1. Minibatches of size 100 were used for both pretraining and dropout finetuning. We tried several network architectures by varying the number of input frames (15 and 31), number of layers in the neural network (3, 4 and 5) and the number of hidden units in each layer (2000 and 4000). Figure  shows the validation error curves for a number of these combinations. Using dropout consistently leads to lower error rates.

## Pretraining

For all our experiments on TIMIT, we pretrain the neural network with a Deep Belief Network . Since the inputs are real-valued, the first layer was pre-trained as a Gaussian RBM. Visible biases were initialized to zero and weights to random numbers sampled from a zero-mean normal distribution with standard deviation 0.01. The variance of each visible unit was set to 1.0 and not learned. Learning was done by minimizing Contrastive Divergence. Momentum was used to speed up learning. Momentum started at 0.5 and was increased linearly to 0.9 over 20 epochs. A learning rate of 0.001 on the average gradient was used (which was then multiplied by 1-momentum). An L2 weight decay of 0.001 was used. The model was trained for 100 epochs.

All subsequent layers were trained as binary RBMs. A learning rate of 0.01 was used. The visible bias of each unit was initialized to $\log(p/(1-p))$ where $p$ was the mean activation of that unit in the dataset. All other hyper-parameters were set to be the same as those we used for the Gaussian RBM. Each layer was trained for 50 epochs.

## Dropout Finetuning

The pretrained RBMs were used to initialize the weights in a neural network. The network was then finetuned with dropout-backpropagation. Momentum was increased from 0.5 to 0.9 linearly over 10 epochs. A small constant learning rate of 1.0 was used (applied to the average gradient on a minibatch). All other hyperparameters are the same as for MNIST dropout finetuning. The model needs to be run for about 200 epochs to converge. The same network was also finetuned with standard backpropagation using a smaller learning rate of 0.1, keeping all other hyperparameters

Figure  shows the frame classification error and cross-entropy objective value on the training and validation sets. We compare the performance of dropout and standard backpropagation on several network architectures and input representations. Dropout consistently achieves lower error and cross-entropy. It significantly controls overfitting, making the method robust to choices of network architecture. It allows much larger nets to be trained and removes the need for early stopping. We also observed that the final error obtained by the model is not very sensitive to the choice of learning rate and momentum.

# Experiments on Reuters

Reuters Corpus Volume I (RCV1-v2) is an archive of 804,414 newswire stories that have been manually categorized into 103 topics[^4]. The corpus covers four major groups: corporate/industrial, economics, government/social, and markets. Sample topics include Energy Markets, Accounts/Earnings, Government Borrowings, Disasters and Accidents, Interbank Markets, Legal/Judicial, Production/Services, etc. The topic classes form a tree which is typically of depth three.

We took the dataset and split it into 63 classes based on the the 63 categories at the second-level of the category tree. We removed 11 categories that did not have any data and one category that had only 4 training examples. We also removed one category that covered a huge chunk (25%) of the examples. This left us with 50 classes and 402,738 documents. We divided the documents into equal-sized training and test sets randomly. Each document was represented using the 2000 most frequent non-stopwords in the dataset.

We trained a neural network using dropout-backpropagation and compared it with standard backpropagation. We used a 2000-2000-1000-50 architecture. The training hyperparameters are same as that in MNIST dropout training (Appendix ). Training was done for 500 epochs.

Figure  shows the training and test set errors as learning progresses. We show two nets - one with a 2000-2000-1000-50 and another with a 2000-1000-1000-50 architecture trained with and without dropout. As in all previous datasets discussed so far, we obtain significant improvements here too. The learning not only results in better generalization, but also proceeds smoothly, without the need for early stopping.

# Tiny Images and CIFAR-10

The Tiny Images dataset contains 80 million $32\times32$ color images collected from the web. The images were found by searching various image search engines for English nouns, so each image comes with a very unreliable label, which is the noun that was used to find it. The CIFAR-10 dataset is a subset of the Tiny Images dataset which contains 60000 images divided among ten classes[^5]. Each class contains 5000 training images and 1000 testing images. The classes are *airplane*, *automobile*, *bird*, *cat*, *deer*, *dog*, *frog*, *horse*, *ship*, and *truck*. The CIFAR-10 dataset was obtained by filtering the Tiny Images dataset to remove images with incorrect labels. The CIFAR-10 images are highly varied, and there is no canonical viewpoint or scale at which the objects appear. The only criteria for including an image were that the image contain one dominant instance of a CIFAR-10 class, and that the object in the image be easily identifiable as belonging to the class indicated by the image label.

# ImageNet

ImageNet is a dataset of millions of labeled images in thousands of categories. The images were collected from the web and labelled by human labellers using Amazon's Mechanical Turk crowd-sourcing tool. In 2010, a subset of roughly 1000 images in each of 1000 classes was the basis of an object recognition competition, a part of the Pascal Visual Object Challenge. This is the version of ImageNet on which we performed our experiments. In all, there are roughly 1.3 million training images, 50000 validation images, and 150000 testing images. This dataset is similar in spirit to the CIFAR-10, but on a much bigger scale. The images are full-resolution, and there are 1000 categories instead of ten. Another difference is that the ImageNet images often contain multiple instances of ImageNet objects, simply due to the sheer number of object classes. For this reason, even a human would have difficulty approaching perfect accuracy on this dataset. For our experiments we resized all images to $256\times256$ pixels.

# Convolutional Neural Networks

Our models for CIFAR-10 and ImageNet are deep, feed-forward convolutional neural networks (CNNs). Feed-forward neural networks are models which consist of several layers of "neurons", where each neuron in a given layer applies a linear filter to the outputs of the neurons in the previous layer. Typically, a scalar bias is added to the filter output and a nonlinear activation function is applied to the result before the neuron's output is passed to the next layer. The linear filters and biases are referred to as *weights*, and these are the parameters of the network that are learned from the training data.

CNNs differ from ordinary neural networks in several ways. First, neurons in a CNN are organized topographically into a bank that reflects the organization of dimensions in the input data. So for images, the neurons are laid out on a 2D grid. Second, neurons in a CNN apply filters which are local in extent and which are centered at the neuron's location in the topographic organization. This is reasonable for datasets where we expect the dependence of input dimensions to be a decreasing function of distance, which is the case for pixels in natural images. In particular, we expect that useful clues to the identity of the object in an input image can be found by examining small local neighborhoods of the image. Third, all neurons in a bank apply the same filter, but as just mentioned, they apply it at different locations in the input image. This is reasonable for datasets with roughly stationary statistics, such as natural images. We expect that the same kinds of structures can appear at all positions in an input image, so it is reasonable to treat all positions equally by filtering them in the same way. In this way, a bank of neurons in a CNN applies a convolution operation to its input. A single layer in a CNN typically has multiple banks of neurons, each performing a convolution with a different filter. These banks of neurons become distinct input channels into the next layer. The distance, in pixels, between the boundaries of the receptive fields of neighboring neurons in a convolutional bank determines the *stride* with which the convolution operation is applied. Larger strides imply fewer neurons per bank. Our models use a stride of one pixel unless otherwise noted.

One important consequence of this convolutional shared-filter architecture is a drastic reduction in the number of parameters relative to a neural net in which all neurons apply different filters. This reduces the net's representational capacity, but it also reduces its capacity to overfit, so dropout is far less advantageous in convolutional layers.

## Pooling

CNNs typically also feature "pooling" layers which summarize the activities of local patches of neurons in convolutional layers. Essentially, a pooling layer takes as input the output of a convolutional layer and subsamples it. A pooling layer consists of pooling units which are laid out topographically and connected to a local neighborhood of convolutional unit outputs from the same bank. Each pooling unit then computes some function of the bank's output in that neighborhood. Typical functions are maximum and average. Pooling layers with such units are called max-pooling and average-pooling layers, respectively. The pooling units are usually spaced at least several pixels apart, so that there are fewer total pooling units than there are convolutional unit outputs in the previous layer. Making this spacing smaller than the size of the neighborhood that the pooling units summarize produces *overlapping pooling.* This variant makes the pooling layer produce a coarse coding of the convolutional unit outputs, which we have found to aid generalization in our experiments. We refer to this spacing as the *stride* between pooling units, analogously to the stride between convolutional units. Pooling layers introduce a level of local translation invariance to the network, which improves generalization. They are the analogues of *complex cells* in the mammalian visual cortex, which pool activities of multiple simple cells. These cells are known to exhibit similar phase-invariance properties.

## Local response normalization

Our networks also include response normalization layers. This type of layer encourages competition for large activations among neurons belonging to different banks. In particular, the activity $a_{x,y}^{i}$ of a neuron in bank $i$ at position $(x,y)$ in the topographic organization is divided by $$\left(1+\alpha\sum_{j=i-N/2}^{i+N/2}(a_{x,y}^{j})^{2}\right)^{\beta}$$ where the sum runs over $N$ "adjacent" banks of neurons at *the same position in the topographic organization*. The ordering of the banks is of course arbitrary and determined before training begins. Response normalization layers implement a form of lateral inhibition found in real neurons. The constants $N,\alpha$, and $\beta$ are hyper-parameters whose values are determined using a validation set.

## Neuron nonlinearities

All of the neurons in our networks utilize the max-with-zero nonlinearity. That is, their output is computed as $a_{x,y}^{i}=\max(0,z_{x,y}^{i})$ where $z_{x,y}^{i}$ is the total input to the neuron (equivalently, the output of the neuron's linear filter added to the bias). This nonlinearity has several advantages over traditional saturating neuron models, including a significant reduction in the training time required to reach a given error rate. This nonlinearity also reduces the need for contrast-normalization and similar data pre-processing schemes, because neurons with this nonlinearity do not saturate – their activities simply scale up when presented with unusually large input values. Consequently, the only data pre-processing step which we take is to subtract the mean activity from each pixel, so that the data is centered. So we train our networks on the (centered) raw RGB values of the pixels.

## Objective function

Our networks maximize the multinomial logistic regression objective, which is equivalent to minimizing the average across training cases of the cross-entropy between the true label distribution and the model's predicted label distribution.

## Weight initialization

We initialize the weights in our model from a zero-mean normal distribution with a variance set high enough to produce positive inputs into the neurons in each layer. This is a slightly tricky point when using the max-with-zero nonlinearity. If the input to a neuron is always negative, no learning will take place because its output will be uniformly zero, as will the derivative of its output with respect to its input. Therefore it's important to initialize the weights from a distribution with a sufficiently large variance such that all neurons are likely to get positive inputs at least occasionally. In practice, we simply try different variances until we find an initialization that works. It usually only takes a few attempts. We also find that initializing the biases of the neurons in the hidden layers with some positive constant (1 in our case) helps get learning off the ground, for the same reason.

## Training

We train our models using stochastic gradient descent with a batch size of 128 examples and momentum of 0.9. Therefore the update rule for weight $w$ is $$\begin{aligned}
v_{i+1} & = & 0.9 v_{i}+\epsilon<\frac{\partial E}{\partial w_{i}}>_{i}\\
w_{i+1} & = & w_{i}+v_{i+1}
\end{aligned}$$ where $i$ is the iteration index, $v$ is a momentum variable, $\epsilon$ is the learning rate, and $<\frac{\partial E}{\partial w_{i}}>_{i}$ is the average over the $i^{th}$ batch of the derivative of the objective with respect to $w_{i}$. We use the publicly available `cuda-convnet` package to train all of our models on a single NVIDIA GTX 580 GPU. Training on CIFAR-10 takes roughly 90 minutes. Training on ImageNet takes roughly four days with dropout and two days without.

## Learning rates

We use an equal learning rate for each layer, whose value we determine heuristically as the largest power of ten that produces reductions in the objective function. In practice it is typically of the order $10^{-2}$ or $10^{-3}$. We reduce the learning rate twice by a factor of ten shortly before terminating training.

# Models for CIFAR-10

Our model for CIFAR-10 without dropout is a CNN with three convolutional layers. Pooling layers follow all three. All of the pooling layers summarize a $3\times3$ neighborhood and use a stride of 2. The pooling layer which follows the first convolutional layer performs max-pooling, while the remaining pooling layers perform average-pooling. Response normalization layers follow the first two pooling layers, with $N=9$, $\alpha=0.001$, and $\beta=0.75$. The upper-most pooling layer is connected to a ten-unit softmax layer which outputs a probability distribution over class labels. All convolutional layers have 64 filter banks and use a filter size of $5\times5$ (times the number of channels in the preceding layer).

Our model for CIFAR-10 with dropout is similar, but because dropout imposes a strong regularization on the network, we are able to use more parameters. Therefore we add a fourth weight layer, which takes its input from the third pooling layer. This weight layer is *locally-connected but not convolutional.* It is like a convolutional layer in which filters in the same bank do not share weights. This layer contains 16 banks of filters of size $3\times3$. This is the layer in which we use 50% dropout. The softmax layer takes its input from this fourth weight layer.

# Models for ImageNet

Our model for ImageNet with dropout is a CNN which is trained on $224\times224$ patches randomly extracted from the $256\times256$ images, as well as their horizontal reflections. This is a form of data augmentation that reduces the network's capacity to overfit the training data and helps generalization. The network contains seven weight layers. The first five are convolutional, while the last two are globally-connected. Max-pooling layers follow the first, second, and fifth convolutional layers. All of the pooling layers summarize a $3\times3$ neighborhood and use a stride of 2. Response-normalization layers follow the first and second pooling layers. The first convolutional layer has 64 filter banks with $11\times11$ filters which it applies with a stride of 4 pixels (this is the distance between neighboring neurons in a bank). The second convolutional layer has 256 filter banks with $5\times5$ filters. This layer takes two inputs. The first input to this layer is the (pooled and response-normalized) output of the first convolutional layer. The 256 banks in this layer are divided arbitrarily into groups of 64, and each group connects to a unique random 16 channels from the first convolutional layer. The second input to this layer is a subsampled version of the original image ($56\times56$), which is filtered by this layer with a stride of 2 pixels. The two maps resulting from filtering the two inputs are summed element-wise (they have exactly the same dimensions) and a max-with-zero nonlinearity is applied to the sum in the usual way. The third, fourth, and fifth convolutional layers are connected to one another without any intervening pooling or normalization layers, but the max-with-zero nonlinearity is applied at each layer after linear filtering. The third convolutional layer has 512 filter banks divided into groups of 32, each group connecting to a unique random subset of 16 channels produced by the (pooled, normalized) outputs of the second convolutional layer. The fourth and fifth convolutional layers similarly have 512 filter banks divided into groups of 32, each group connecting to a unique random subset of 32 channels produced by the layer below. The next two weight layers are globally-connected, with 4096 neurons each. In these last two layers we use 50% dropout. Finally, the output of the last globally-connected layer is fed to a 1000-way softmax which produces a distribution over the 1000 class labels. We test our model by averaging the prediction of the net on ten $224\times224$ patches of the $256\times256$ input image: the center patch, the four corner patches, and their horizontal reflections. Even though we make ten passes of each image at test time, we are able to run our system in real-time.

Our model for ImageNet without dropout is similar, but without the two globally-connected layers which create serious overfitting when used without dropout.

In order to achieve state-of-the-art performance on the validation set, we found it necessary to use the very complicated network architecture described above. Fortunately, the complexity of this architecture is not the main point of our paper. What we wanted to demonstrate is that dropout is a significant help even for the very complex neural nets that have been developed by the joint efforts of many groups over many years to be really good at object recognition. This is clearly demonstrated by the fact that using non-convolutional higher layers with a lot of parameters leads to a big improvement with dropout but makes things worse without dropout.

[^1]: For code see <a href="http://www.cs.toronto.edu/~hinton/MatlabForSciencePaper.html " class="uri">http://www.cs.toronto.edu/~hinton/MatlabForSciencePaper.html </a>

[^2]: For code see <a href="http://www.utstat.toronto.edu/~rsalakhu/DBM.html " class="uri">http://www.utstat.toronto.edu/~rsalakhu/DBM.html </a>

[^3]: <a href=" http://kaldi.sourceforge.net " class="uri"> http://kaldi.sourceforge.net </a>

[^4]: The corpus is available at http://www.ai.mit.edu/projects/jmlr/papers/volume5/lewis04a/lyrl2004_rcv1v2_README.htm

[^5]: The CIFAR dataset is available at http://www.cs.toronto.edu/$\sim$kriz/cifar.html.