WIP/DO NOT MERGE: Categorical improvements
State: closed

This is a PR to make discussing the doc changes easier. See https://github.com/pydata/pandas/pull/7217 for the main PR

TODO List: now in #7217
## The Docs (updated 1st july, 4pm CEST)

 <div class="section" id="categorical">
<span id="id1"></span><h1>Categorical<a class="headerlink" href="#categorical" title="Permalink to this headline">¶</a></h1>

<div class="versionadded">
<p><span class="versionmodified">New in version 0.15.</span></p>
</div>

<div class="admonition note">
<p class="first admonition-title">Note</p>
<p class="last">While there was in <cite>pandas.Categorical</cite> in earlier versions, the ability to use
<cite>Categorical</cite> data in <cite>Series</cite> and <cite>DataFrame</cite> is new.</p>
</div>

<p>This is a short introduction to pandas <cite>Categorical</cite> type, including a short comparison with R&#8217;s
<cite>factor</cite>.</p>

<p><cite>Categoricals</cite> are a pandas data type, which correspond to categorical variables in
statistics: a variable, which can take on only a limited, and usually fixed,
number of possible values (commonly called <cite>levels</cite>). Examples are gender, social class,
blood types, country affiliations, observation time or ratings via Likert scales.</p>

<p>In contrast to statistical categorical variables, a <cite>Categorical</cite> might have an order (e.g.
&#8216;strongly agree&#8217; vs &#8216;agree&#8217; or &#8216;first observation&#8217; vs. &#8216;second observation&#8217;), but numerical
operations (additions, divisions, ...) are not possible.</p>

<p>All values of the <cite>Categorical</cite> are either in <cite>levels</cite> or <cite>np.nan</cite>. Order is defined by
the order of the <cite>levels</cite>, not lexical order of the values. Internally, the data structure
consists of a levels array and an integer array of level_codes which point to the real value in the
levels array.</p>

<p><cite>Categoricals</cite> are useful in the following cases:</p>

<ul class="simple">
<li>A string variable consisting of only a few different values. Converting such a string
variable to a categorical variable will save some memory.</li>
<li>The lexical order of a variable is not the same as the logical order (&#8220;one&#8221;, &#8220;two&#8221;, &#8220;three&#8221;).
By converting to a categorical and specifying an order on the levels, sorting and
min/max will use the logical order instead of the lexical order.</li>
<li>As a signal to other python libraries that this column should be treated as a categorical
variable (e.g. to use suitable statistical methods or plot types)</li>
</ul>

<p>See also the <a class="reference internal" href="api.html#api-categorical"><em>API docs on Categoricals</em></a>.</p>

<div class="section" id="object-creation">
<h2>Object Creation<a class="headerlink" href="#object-creation" title="Permalink to this headline">¶</a></h2>
<p>Categorical <cite>Series</cite> or columns in a <cite>DataFrame</cite> can be crated in several ways:</p>
<p>By passing a <cite>Categorical</cite> object to a <cite>Series</cite> or assigning it to a <cite>DataFrame</cite>:</p>
<div class="highlight-ipython"><div class="highlight"><pre><span class="gp">In [1]: </span><span class="n">raw_cat</span> <span class="o">=</span> <span class="n">pd</span><span class="o">.</span><span class="n">Categorical</span><span class="p">([</span><span class="s">&quot;a&quot;</span><span class="p">,</span><span class="s">&quot;b&quot;</span><span class="p">,</span><span class="s">&quot;c&quot;</span><span class="p">,</span><span class="s">&quot;a&quot;</span><span class="p">])</span>

<span class="gp">In [2]: </span><span class="n">s</span> <span class="o">=</span> <span class="n">pd</span><span class="o">.</span><span class="n">Series</span><span class="p">(</span><span class="n">raw_cat</span><span class="p">)</span>

<span class="gp">In [3]: </span><span class="n">s</span>
<span class="gr">Out[3]: </span>
<span class="go">0    a</span>
<span class="go">1    b</span>
<span class="go">2    c</span>
<span class="go">3    a</span>
<span class="go">dtype: category</span>

<span class="gp">In [4]: </span><span class="n">df</span> <span class="o">=</span> <span class="n">pd</span><span class="o">.</span><span class="n">DataFrame</span><span class="p">({</span><span class="s">&quot;A&quot;</span><span class="p">:[</span><span class="s">&quot;a&quot;</span><span class="p">,</span><span class="s">&quot;b&quot;</span><span class="p">,</span><span class="s">&quot;c&quot;</span><span class="p">,</span><span class="s">&quot;a&quot;</span><span class="p">]})</span>

<span class="gp">In [5]: </span><span class="n">df</span><span class="p">[</span><span class="s">&quot;B&quot;</span><span class="p">]</span> <span class="o">=</span> <span class="n">raw_cat</span>

<span class="gp">In [6]: </span><span class="n">df</span>
<span class="gr">Out[6]: </span>
<span class="go">   A  B</span>
<span class="go">0  a  a</span>
<span class="go">1  b  b</span>
<span class="go">2  c  c</span>
<span class="go">3  a  a</span>
</pre></div>
</div>

<p>By converting an existing <cite>Series</cite> or column to a <tt class="docutils literal"><span class="pre">category</span></tt> type:</p>

<div class="highlight-ipython"><div class="highlight"><pre><span class="gp">In [7]: </span><span class="n">df</span> <span class="o">=</span> <span class="n">pd</span><span class="o">.</span><span class="n">DataFrame</span><span class="p">({</span><span class="s">&quot;A&quot;</span><span class="p">:[</span><span class="s">&quot;a&quot;</span><span class="p">,</span><span class="s">&quot;b&quot;</span><span class="p">,</span><span class="s">&quot;c&quot;</span><span class="p">,</span><span class="s">&quot;a&quot;</span><span class="p">]})</span>

<span class="gp">In [8]: </span><span class="n">df</span><span class="p">[</span><span class="s">&quot;B&quot;</span><span class="p">]</span> <span class="o">=</span> <span class="n">df</span><span class="p">[</span><span class="s">&quot;A&quot;</span><span class="p">]</span><span class="o">.</span><span class="n">astype</span><span class="p">(</span><span class="s">&#39;category&#39;</span><span class="p">)</span>

<span class="gp">In [9]: </span><span class="n">df</span>
<span class="gr">Out[9]: </span>
<span class="go">   A  B</span>
<span class="go">0  a  a</span>
<span class="go">1  b  b</span>
<span class="go">2  c  c</span>
<span class="go">3  a  a</span>
</pre></div>
</div>

<p>By using some special functions:</p>

<div class="highlight-ipython"><div class="highlight"><pre><span class="gp">In [10]: </span><span class="n">df</span> <span class="o">=</span> <span class="n">pd</span><span class="o">.</span><span class="n">DataFrame</span><span class="p">({</span><span class="s">&#39;value&#39;</span><span class="p">:</span> <span class="n">np</span><span class="o">.</span><span class="n">random</span><span class="o">.</span><span class="n">randint</span><span class="p">(</span><span class="mi">0</span><span class="p">,</span> <span class="mi">100</span><span class="p">,</span> <span class="mi">20</span><span class="p">)})</span>

<span class="gp">In [11]: </span><span class="n">labels</span> <span class="o">=</span> <span class="p">[</span> <span class="s">&quot;{0} - {1}&quot;</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">i</span><span class="p">,</span> <span class="n">i</span> <span class="o">+</span> <span class="mi">9</span><span class="p">)</span> <span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="mi">0</span><span class="p">,</span> <span class="mi">100</span><span class="p">,</span> <span class="mi">10</span><span class="p">)</span> <span class="p">]</span>

<span class="gp">In [12]: </span><span class="n">df</span><span class="p">[</span><span class="s">&#39;group&#39;</span><span class="p">]</span> <span class="o">=</span> <span class="n">pd</span><span class="o">.</span><span class="n">cut</span><span class="p">(</span><span class="n">df</span><span class="o">.</span><span class="n">value</span><span class="p">,</span> <span class="nb">range</span><span class="p">(</span><span class="mi">0</span><span class="p">,</span> <span class="mi">105</span><span class="p">,</span> <span class="mi">10</span><span class="p">),</span> <span class="n">right</span><span class="o">=</span><span class="bp">False</span><span class="p">,</span> <span class="n">labels</span><span class="o">=</span><span class="n">labels</span><span class="p">)</span>

<span class="gp">In [13]: </span><span class="n">df</span><span class="o">.</span><span class="n">head</span><span class="p">(</span><span class="mi">10</span><span class="p">)</span>
<span class="gr">Out[13]: </span>
<span class="go">   value    group</span>
<span class="go">0     65  60 - 69</span>
<span class="go">1     49  40 - 49</span>
<span class="go">2     56  50 - 59</span>
<span class="go">3     43  40 - 49</span>
<span class="go">4     43  40 - 49</span>
<span class="go">5     91  90 - 99</span>
<span class="go">6     32  30 - 39</span>
<span class="go">7     87  80 - 89</span>
<span class="go">8     36  30 - 39</span>
<span class="go">9      8    0 - 9</span>
</pre></div>
</div>

<p><cite>Categoricals</cite> have a specific <tt class="docutils literal"><span class="pre">category</span></tt> <a class="reference internal" href="basics.html#basics-dtypes"><em>dtype</em></a>:</p>

<div class="highlight-ipython"><div class="highlight"><pre><span class="gp">In [14]: </span><span class="n">df</span><span class="o">.</span><span class="n">dtypes</span>
<span class="gr">Out[14]: </span>
<span class="go">value       int32</span>
<span class="go">group    category</span>
<span class="go">dtype: object</span>
</pre></div>
</div>

<div class="admonition note">
<p class="first admonition-title">Note</p>
<p class="last">In contrast to R&#8217;s <cite>factor</cite> function, a <cite>Categorical</cite> is not converting input values to
string and levels will end up the same data type as the original values.</p>
</div>

<div class="admonition note">
<p class="first admonition-title">Note</p>
<p class="last">I contrast to R&#8217;s <cite>factor</cite> function, there is currently no way to assign/change labels at
creation time. Use <cite>levels</cite> to change the levels after creation time.</p>
</div>

<p>To get back to the original Series or <cite>numpy</cite> array, use <tt class="docutils literal"><span class="pre">Series.astype(original_dtype)</span></tt> or
<tt class="docutils literal"><span class="pre">np.asarray(categorical)</span></tt>:</p>

<div class="highlight-ipython"><div class="highlight"><pre><span class="gp">In [15]: </span><span class="n">s</span> <span class="o">=</span> <span class="n">pd</span><span class="o">.</span><span class="n">Series</span><span class="p">([</span><span class="s">&quot;a&quot;</span><span class="p">,</span><span class="s">&quot;b&quot;</span><span class="p">,</span><span class="s">&quot;c&quot;</span><span class="p">,</span><span class="s">&quot;a&quot;</span><span class="p">])</span>

<span class="gp">In [16]: </span><span class="n">s</span>
<span class="gr">Out[16]: </span>
<span class="go">0    a</span>
<span class="go">1    b</span>
<span class="go">2    c</span>
<span class="go">3    a</span>
<span class="go">dtype: object</span>

<span class="gp">In [17]: </span><span class="n">s2</span> <span class="o">=</span> <span class="n">s</span><span class="o">.</span><span class="n">astype</span><span class="p">(</span><span class="s">&#39;category&#39;</span><span class="p">)</span>

<span class="gp">In [18]: </span><span class="n">s2</span>
<span class="gr">Out[18]: </span>
<span class="go">0    a</span>
<span class="go">1    b</span>
<span class="go">2    c</span>
<span class="go">3    a</span>
<span class="go">dtype: category</span>

<span class="gp">In [19]: </span><span class="n">s3</span> <span class="o">=</span> <span class="n">s2</span><span class="o">.</span><span class="n">astype</span><span class="p">(</span><span class="s">&#39;string&#39;</span><span class="p">)</span>

<span class="gp">In [20]: </span><span class="n">s3</span>
<span class="gr">Out[20]: </span>
<span class="go">0    a</span>
<span class="go">1    b</span>
<span class="go">2    c</span>
<span class="go">3    a</span>
<span class="go">dtype: object</span>

<span class="gp">In [21]: </span><span class="n">np</span><span class="o">.</span><span class="n">asarray</span><span class="p">(</span><span class="n">s2</span><span class="o">.</span><span class="n">cat</span><span class="p">)</span>
<span class="gr">Out[21]: </span><span class="n">array</span><span class="p">([</span><span class="s">&#39;a&#39;</span><span class="p">,</span> <span class="s">&#39;b&#39;</span><span class="p">,</span> <span class="s">&#39;c&#39;</span><span class="p">,</span> <span class="s">&#39;a&#39;</span><span class="p">],</span> <span class="n">dtype</span><span class="o">=</span><span class="nb">object</span><span class="p">)</span>
</pre></div>
</div>

</div>

<div class="section" id="working-with-levels">
<h2>Working with levels<a class="headerlink" href="#working-with-levels" title="Permalink to this headline">¶</a></h2>
<p><cite>Categoricals</cite> have a <cite>levels</cite> property, which list their possible values. If you don&#8217;t
manually specify levels, they are inferred from the passed in values. <cite>Series</cite> of type
<tt class="docutils literal"><span class="pre">category</span></tt> expose the same interface via their <cite>cat</cite> property.</p>
<div class="highlight-ipython"><div class="highlight"><pre><span class="gp">In [22]: </span><span class="n">raw_cat</span> <span class="o">=</span> <span class="n">pd</span><span class="o">.</span><span class="n">Categorical</span><span class="p">([</span><span class="s">&quot;a&quot;</span><span class="p">,</span><span class="s">&quot;b&quot;</span><span class="p">,</span><span class="s">&quot;c&quot;</span><span class="p">,</span><span class="s">&quot;a&quot;</span><span class="p">])</span>

<span class="gp">In [23]: </span><span class="n">raw_cat</span><span class="o">.</span><span class="n">levels</span>
<span class="gr">Out[23]: </span><span class="n">Index</span><span class="p">([</span><span class="s">u&#39;a&#39;</span><span class="p">,</span> <span class="s">u&#39;b&#39;</span><span class="p">,</span> <span class="s">u&#39;c&#39;</span><span class="p">],</span> <span class="n">dtype</span><span class="o">=</span><span class="s">&#39;object&#39;</span><span class="p">)</span>

<span class="gp">In [24]: </span><span class="n">raw_cat</span><span class="o">.</span><span class="n">ordered</span>
<span class="gr">Out[24]: </span><span class="bp">True</span>

<span class="c"># Series of type &quot;category&quot; also expose these interface via the .cat property:</span>
<span class="gp">In [25]: </span><span class="n">s</span> <span class="o">=</span> <span class="n">pd</span><span class="o">.</span><span class="n">Series</span><span class="p">(</span><span class="n">raw_cat</span><span class="p">)</span>

<span class="gp">In [26]: </span><span class="n">s</span><span class="o">.</span><span class="n">cat</span><span class="o">.</span><span class="n">levels</span>
<span class="gr">Out[26]: </span><span class="n">Index</span><span class="p">([</span><span class="s">u&#39;a&#39;</span><span class="p">,</span> <span class="s">u&#39;b&#39;</span><span class="p">,</span> <span class="s">u&#39;c&#39;</span><span class="p">],</span> <span class="n">dtype</span><span class="o">=</span><span class="s">&#39;object&#39;</span><span class="p">)</span>

<span class="gp">In [27]: </span><span class="n">s</span><span class="o">.</span><span class="n">cat</span><span class="o">.</span><span class="n">ordered</span>
<span class="gr">Out[27]: </span><span class="bp">True</span>
</pre></div>
</div>

<div class="admonition note">
<p class="first admonition-title">Note</p>
<p class="last">New <cite>Categorical</cite> are automatically ordered if the passed in values are sortable or a
<cite>levels</cite> argument is supplied. This is a difference to R&#8217;s <cite>factors</cite>, which are unordered
unless explicitly told to be ordered (<tt class="docutils literal"><span class="pre">ordered=TRUE</span></tt>).</p>
</div>

<p>It&#8217;s also possible to pass in the levels in a specific order:</p>

<div class="highlight-ipython"><div class="highlight"><pre><span class="gp">In [28]: </span><span class="n">raw_cat</span> <span class="o">=</span> <span class="n">pd</span><span class="o">.</span><span class="n">Categorical</span><span class="p">([</span><span class="s">&quot;a&quot;</span><span class="p">,</span><span class="s">&quot;b&quot;</span><span class="p">,</span><span class="s">&quot;c&quot;</span><span class="p">,</span><span class="s">&quot;a&quot;</span><span class="p">],</span> <span class="n">levels</span><span class="o">=</span><span class="p">[</span><span class="s">&quot;c&quot;</span><span class="p">,</span><span class="s">&quot;b&quot;</span><span class="p">,</span><span class="s">&quot;a&quot;</span><span class="p">])</span>

<span class="gp">In [29]: </span><span class="n">s</span> <span class="o">=</span> <span class="n">pd</span><span class="o">.</span><span class="n">Series</span><span class="p">(</span><span class="n">raw_cat</span><span class="p">)</span>

<span class="gp">In [30]: </span><span class="n">s</span><span class="o">.</span><span class="n">cat</span><span class="o">.</span><span class="n">levels</span>
<span class="gr">Out[30]: </span><span class="n">Index</span><span class="p">([</span><span class="s">u&#39;c&#39;</span><span class="p">,</span> <span class="s">u&#39;b&#39;</span><span class="p">,</span> <span class="s">u&#39;a&#39;</span><span class="p">],</span> <span class="n">dtype</span><span class="o">=</span><span class="s">&#39;object&#39;</span><span class="p">)</span>

<span class="gp">In [31]: </span><span class="n">s</span><span class="o">.</span><span class="n">cat</span><span class="o">.</span><span class="n">ordered</span>
<span class="gr">Out[31]: </span><span class="bp">True</span>
</pre></div>
</div>

<div class="admonition note">
<p class="first admonition-title">Note</p>
<p class="last">Passing in a <cite>levels</cite> argument implies <tt class="docutils literal"><span class="pre">ordered=True</span></tt>.</p>
</div>

<p>Any value omitted in the levels argument will be replaced by <cite>np.nan</cite>:</p>

<div class="highlight-ipython"><div class="highlight"><pre><span class="gp">In [32]: </span><span class="n">raw_cat</span> <span class="o">=</span> <span class="n">pd</span><span class="o">.</span><span class="n">Categorical</span><span class="p">([</span><span class="s">&quot;a&quot;</span><span class="p">,</span><span class="s">&quot;b&quot;</span><span class="p">,</span><span class="s">&quot;c&quot;</span><span class="p">,</span><span class="s">&quot;a&quot;</span><span class="p">],</span> <span class="n">levels</span><span class="o">=</span><span class="p">[</span><span class="s">&quot;a&quot;</span><span class="p">,</span><span class="s">&quot;b&quot;</span><span class="p">])</span>

<span class="gp">In [33]: </span><span class="n">s</span> <span class="o">=</span> <span class="n">pd</span><span class="o">.</span><span class="n">Series</span><span class="p">(</span><span class="n">raw_cat</span><span class="p">)</span>

<span class="gp">In [34]: </span><span class="n">s</span><span class="o">.</span><span class="n">cat</span><span class="o">.</span><span class="n">levels</span>
<span class="gr">Out[34]: </span><span class="n">Index</span><span class="p">([</span><span class="s">u&#39;a&#39;</span><span class="p">,</span> <span class="s">u&#39;b&#39;</span><span class="p">],</span> <span class="n">dtype</span><span class="o">=</span><span class="s">&#39;object&#39;</span><span class="p">)</span>

<span class="gp">In [35]: </span><span class="n">s</span>
<span class="gr">Out[35]: </span>
<span class="go">0      a</span>
<span class="go">1      b</span>
<span class="go">2    NaN</span>
<span class="go">3      a</span>
<span class="go">dtype: category</span>
</pre></div>
</div>

<p>Renaming levels is done by assigning new values to the <tt class="docutils literal"><span class="pre">Category.levels</span></tt> or
<tt class="docutils literal"><span class="pre">Series.cat.levels</span></tt> property:</p>

<div class="highlight-ipython"><div class="highlight"><pre><span class="gp">In [36]: </span><span class="n">s</span> <span class="o">=</span> <span class="n">pd</span><span class="o">.</span><span class="n">Series</span><span class="p">(</span><span class="n">pd</span><span class="o">.</span><span class="n">Categorical</span><span class="p">([</span><span class="s">&quot;a&quot;</span><span class="p">,</span><span class="s">&quot;b&quot;</span><span class="p">,</span><span class="s">&quot;c&quot;</span><span class="p">,</span><span class="s">&quot;a&quot;</span><span class="p">]))</span>

<span class="gp">In [37]: </span><span class="n">s</span>
<span class="gr">Out[37]: </span>
<span class="go">0    a</span>
<span class="go">1    b</span>
<span class="go">2    c</span>
<span class="go">3    a</span>
<span class="go">dtype: category</span>

<span class="gp">In [38]: </span><span class="n">s</span><span class="o">.</span><span class="n">cat</span><span class="o">.</span><span class="n">levels</span> <span class="o">=</span> <span class="p">[</span><span class="s">&quot;Group </span><span class="si">%s</span><span class="s">&quot;</span> <span class="o">%</span> <span class="n">g</span> <span class="k">for</span> <span class="n">g</span> <span class="ow">in</span> <span class="n">s</span><span class="o">.</span><span class="n">cat</span><span class="o">.</span><span class="n">levels</span><span class="p">]</span>

<span class="gp">In [39]: </span><span class="n">s</span>
<span class="gr">Out[39]: </span>
<span class="go">0    Group a</span>
<span class="go">1    Group b</span>
<span class="go">2    Group c</span>
<span class="go">3    Group a</span>
<span class="go">dtype: category</span>

<span class="gp">In [40]: </span><span class="n">s</span><span class="o">.</span><span class="n">cat</span><span class="o">.</span><span class="n">levels</span> <span class="o">=</span> <span class="p">[</span><span class="mi">1</span><span class="p">,</span><span class="mi">2</span><span class="p">,</span><span class="mi">3</span><span class="p">]</span>

<span class="gp">In [41]: </span><span class="n">s</span>
<span class="gr">Out[41]: </span>
<span class="go">0    1</span>
<span class="go">1    2</span>
<span class="go">2    3</span>
<span class="go">3    1</span>
<span class="go">dtype: category</span>
</pre></div>
</div>

<div class="admonition note">
<p class="first admonition-title">Note</p>
<p class="last">I contrast to R&#8217;s <cite>factor</cite> function, a <cite>Categorical</cite> can have levels of other types than
string.</p>
</div>

<p>Levels must be unique or a <cite>ValueError</cite> is raised:</p>

<div class="highlight-ipython"><div class="highlight"><pre><span class="gp">In [42]: </span><span class="k">try</span><span class="p">:</span>
<span class="gp">   ....:</span>     <span class="n">s</span><span class="o">.</span><span class="n">cat</span><span class="o">.</span><span class="n">levels</span> <span class="o">=</span> <span class="p">[</span><span class="mi">1</span><span class="p">,</span><span class="mi">1</span><span class="p">,</span><span class="mi">1</span><span class="p">]</span>
<span class="gp">   ....:</span> <span class="k">except</span> <span class="ne">ValueError</span> <span class="k">as</span> <span class="n">e</span><span class="p">:</span>
<span class="gp">   ....:</span>     <span class="k">print</span><span class="p">(</span><span class="s">&quot;ValueError: &quot;</span> <span class="o">+</span> <span class="nb">str</span><span class="p">(</span><span class="n">e</span><span class="p">))</span>
<span class="gp">   ....:</span> 
<span class="go">ValueError: Categorical levels must be unique</span>
</pre></div>
</div>

<p>Appending a level can be done by assigning a levels list longer than the current levels:</p>

<div class="highlight-ipython"><div class="highlight"><pre><span class="gp">In [43]: </span><span class="n">s</span><span class="o">.</span><span class="n">cat</span><span class="o">.</span><span class="n">levels</span> <span class="o">=</span> <span class="p">[</span><span class="mi">1</span><span class="p">,</span><span class="mi">2</span><span class="p">,</span><span class="mi">3</span><span class="p">,</span><span class="mi">4</span><span class="p">]</span>

<span class="gp">In [44]: </span><span class="n">s</span><span class="o">.</span><span class="n">cat</span><span class="o">.</span><span class="n">levels</span>
<span class="gr">Out[44]: </span><span class="n">Int64Index</span><span class="p">([</span><span class="mi">1</span><span class="p">,</span> <span class="mi">2</span><span class="p">,</span> <span class="mi">3</span><span class="p">,</span> <span class="mi">4</span><span class="p">],</span> <span class="n">dtype</span><span class="o">=</span><span class="s">&#39;int64&#39;</span><span class="p">)</span>

<span class="gp">In [45]: </span><span class="n">s</span>
<span class="gr">Out[45]: </span>
<span class="go">0    1</span>
<span class="go">1    2</span>
<span class="go">2    3</span>
<span class="go">3    1</span>
<span class="go">dtype: category</span>
</pre></div>
</div>

<p>Removing a level is also possible, but only the last level(s) can be removed by assigning a
shorter list than current levels. Values which are omitted are replaced by <cite>np.nan</cite>.</p>

<div class="highlight-ipython"><div class="highlight"><pre><span class="gp">In [46]: </span><span class="n">s</span><span class="o">.</span><span class="n">levels</span> <span class="o">=</span> <span class="p">[</span><span class="mi">1</span><span class="p">,</span><span class="mi">2</span><span class="p">]</span>

<span class="gp">In [47]: </span><span class="n">s</span>
<span class="gr">Out[47]: </span>
<span class="go">0    1</span>
<span class="go">1    2</span>
<span class="go">2    3</span>
<span class="go">3    1</span>
<span class="go">dtype: category</span>
</pre></div>
</div>

<div class="admonition note">
<p class="first admonition-title">Note</p>
<p class="last">It&#8217;s only possible to remove or add a level at the last position. If that&#8217;s not where you want
to remove an old or add a new level, use <tt class="docutils literal"><span class="pre">Category.reorder_levels(new_order)</span></tt> or
<tt class="docutils literal"><span class="pre">Series.cat.reorder_levels(new_order)</span></tt> methods before or after.</p>
</div>

<p>Removing unused levels can also be done:</p>

<div class="highlight-ipython"><div class="highlight"><pre><span class="gp">In [48]: </span><span class="n">raw</span> <span class="o">=</span> <span class="n">pd</span><span class="o">.</span><span class="n">Categorical</span><span class="p">([</span><span class="s">&quot;a&quot;</span><span class="p">,</span><span class="s">&quot;b&quot;</span><span class="p">,</span><span class="s">&quot;a&quot;</span><span class="p">],</span> <span class="n">levels</span><span class="o">=</span><span class="p">[</span><span class="s">&quot;a&quot;</span><span class="p">,</span><span class="s">&quot;b&quot;</span><span class="p">,</span><span class="s">&quot;c&quot;</span><span class="p">,</span><span class="s">&quot;d&quot;</span><span class="p">])</span>

<span class="gp">In [49]: </span><span class="n">c</span> <span class="o">=</span> <span class="n">pd</span><span class="o">.</span><span class="n">Series</span><span class="p">(</span><span class="n">raw</span><span class="p">)</span>

<span class="gp">In [50]: </span><span class="n">raw</span>
<span class="gr">Out[50]: </span>
<span class="go"> a</span>
<span class="go"> b</span>
<span class="go"> a</span>
<span class="go">Levels (4): Index([&#39;a&#39;, &#39;b&#39;, &#39;c&#39;, &#39;d&#39;], dtype=object), ordered</span>

<span class="gp">In [51]: </span><span class="n">raw</span><span class="o">.</span><span class="n">remove_unused_levels</span><span class="p">()</span>

<span class="gp">In [52]: </span><span class="n">raw</span>
<span class="gr">Out[52]: </span>
<span class="go"> a</span>
<span class="go"> b</span>
<span class="go"> a</span>
<span class="go">Levels (2): Index([&#39;a&#39;, &#39;b&#39;], dtype=object), ordered</span>

<span class="gp">In [53]: </span><span class="n">c</span><span class="o">.</span><span class="n">cat</span><span class="o">.</span><span class="n">remove_unused_levels</span><span class="p">()</span>

<span class="gp">In [54]: </span><span class="n">c</span>
<span class="gr">Out[54]: </span>
<span class="go">0    a</span>
<span class="go">1    b</span>
<span class="go">2    a</span>
<span class="go">dtype: category</span>
</pre></div>
</div>

<div class="admonition note">
<p class="first admonition-title">Note</p>
<p class="last">In contrast to R&#8217;s <cite>factor</cite> function, passing a <cite>Categorical</cite> as the sole input to the
<cite>Categorical</cite> constructor will <em>not</em> remove unused levels but create a new <cite>Categorical</cite>
which is equal to the passed in one!</p>
</div>

</div>

<div class="section" id="ordered-or-not">
<h2>Ordered or not...<a class="headerlink" href="#ordered-or-not" title="Permalink to this headline">¶</a></h2>
<p>If a <cite>Categoricals</cite> is ordered (<tt class="docutils literal"><span class="pre">cat.ordered</span> <span class="pre">==</span> <span class="pre">True</span></tt>), then the order of the levels has a
meaning and certain operations are possible. If the categorical is unordered, a <cite>TypeError</cite> is
raised.</p>
<div class="highlight-ipython"><div class="highlight"><pre><span class="gp">In [55]: </span><span class="n">s</span> <span class="o">=</span> <span class="n">pd</span><span class="o">.</span><span class="n">Series</span><span class="p">(</span><span class="n">pd</span><span class="o">.</span><span class="n">Categorical</span><span class="p">([</span><span class="s">&quot;a&quot;</span><span class="p">,</span><span class="s">&quot;b&quot;</span><span class="p">,</span><span class="s">&quot;c&quot;</span><span class="p">,</span><span class="s">&quot;a&quot;</span><span class="p">],</span> <span class="n">ordered</span><span class="o">=</span><span class="bp">False</span><span class="p">))</span>

<span class="gp">In [56]: </span><span class="k">try</span><span class="p">:</span>
<span class="gp">   ....:</span>     <span class="n">s</span><span class="o">.</span><span class="n">sort</span><span class="p">()</span>
<span class="gp">   ....:</span> <span class="k">except</span> <span class="ne">TypeError</span> <span class="k">as</span> <span class="n">e</span><span class="p">:</span>
<span class="gp">   ....:</span>     <span class="k">print</span><span class="p">(</span><span class="s">&quot;TypeError: &quot;</span> <span class="o">+</span> <span class="nb">str</span><span class="p">(</span><span class="n">e</span><span class="p">))</span>
<span class="gp">   ....:</span> 
<span class="go">TypeError: Categorical not ordered</span>

<span class="gp">In [57]: </span><span class="n">s</span> <span class="o">=</span> <span class="n">pd</span><span class="o">.</span><span class="n">Series</span><span class="p">(</span><span class="n">pd</span><span class="o">.</span><span class="n">Categorical</span><span class="p">([</span><span class="s">&quot;a&quot;</span><span class="p">,</span><span class="s">&quot;b&quot;</span><span class="p">,</span><span class="s">&quot;c&quot;</span><span class="p">,</span><span class="s">&quot;a&quot;</span><span class="p">],</span> <span class="n">ordered</span><span class="o">=</span><span class="bp">True</span><span class="p">))</span>

<span class="gp">In [58]: </span><span class="n">s</span><span class="o">.</span><span class="n">sort</span><span class="p">()</span>

<span class="gp">In [59]: </span><span class="n">s</span>
<span class="gr">Out[59]: </span>
<span class="go">0    a</span>
<span class="go">3    a</span>
<span class="go">1    b</span>
<span class="go">2    c</span>
<span class="go">dtype: category</span>

<span class="gp">In [60]: </span><span class="k">print</span><span class="p">(</span><span class="n">s</span><span class="o">.</span><span class="n">min</span><span class="p">(),</span> <span class="n">s</span><span class="o">.</span><span class="n">max</span><span class="p">())</span>
<span class="go">(&#39;a&#39;, &#39;c&#39;)</span>
</pre></div>
</div>

<div class="admonition note">
<p class="first admonition-title">Note</p>
<p class="last"><tt class="docutils literal"><span class="pre">ordered=True</span></tt> is not necessary needed in the second case, as lists of strings are sortable
and so the resulting <cite>Categorical</cite> is ordered.</p>
</div>

<p>Sorting will use the order defined by levels, not any lexical order present on the data type.
This is even true for strings and numeric data:</p>

<div class="highlight-ipython"><div class="highlight"><pre><span class="gp">In [61]: </span><span class="n">s</span> <span class="o">=</span> <span class="n">pd</span><span class="o">.</span><span class="n">Series</span><span class="p">(</span><span class="n">pd</span><span class="o">.</span><span class="n">Categorical</span><span class="p">([</span><span class="mi">1</span><span class="p">,</span><span class="mi">2</span><span class="p">,</span><span class="mi">3</span><span class="p">,</span><span class="mi">1</span><span class="p">]))</span>

<span class="gp">In [62]: </span><span class="n">s</span><span class="o">.</span><span class="n">cat</span><span class="o">.</span><span class="n">levels</span> <span class="o">=</span> <span class="p">[</span><span class="mi">2</span><span class="p">,</span><span class="mi">3</span><span class="p">,</span><span class="mi">1</span><span class="p">]</span>

<span class="gp">In [63]: </span><span class="n">s</span>
<span class="gr">Out[63]: </span>
<span class="go">0    2</span>
<span class="go">1    3</span>
<span class="go">2    1</span>
<span class="go">3    2</span>
<span class="go">dtype: category</span>

<span class="gp">In [64]: </span><span class="n">s</span><span class="o">.</span><span class="n">sort</span><span class="p">()</span>

<span class="gp">In [65]: </span><span class="n">s</span>
<span class="gr">Out[65]: </span>
<span class="go">0    2</span>
<span class="go">3    2</span>
<span class="go">1    3</span>
<span class="go">2    1</span>
<span class="go">dtype: category</span>

<span class="gp">In [66]: </span><span class="k">print</span><span class="p">(</span><span class="n">s</span><span class="o">.</span><span class="n">min</span><span class="p">(),</span> <span class="n">s</span><span class="o">.</span><span class="n">max</span><span class="p">())</span>
<span class="go">(2, 1)</span>
</pre></div>
</div>

<p>Reordering the levels is possible via the <tt class="docutils literal"><span class="pre">Categorical.reorder_levels(new_levels)</span></tt>  or
<tt class="docutils literal"><span class="pre">Series.cat.reorder_levels(new_levels)</span></tt> methods:</p>

<div class="highlight-ipython"><div class="highlight"><pre><span class="gp">In [67]: </span><span class="n">s2</span> <span class="o">=</span> <span class="n">pd</span><span class="o">.</span><span class="n">Series</span><span class="p">(</span><span class="n">pd</span><span class="o">.</span><span class="n">Categorical</span><span class="p">([</span><span class="mi">1</span><span class="p">,</span><span class="mi">2</span><span class="p">,</span><span class="mi">3</span><span class="p">,</span><span class="mi">1</span><span class="p">]))</span>

<span class="gp">In [68]: </span><span class="n">s2</span><span class="o">.</span><span class="n">cat</span><span class="o">.</span><span class="n">reorder_levels</span><span class="p">([</span><span class="mi">2</span><span class="p">,</span><span class="mi">3</span><span class="p">,</span><span class="mi">1</span><span class="p">])</span>

<span class="gp">In [69]: </span><span class="n">s2</span>
<span class="gr">Out[69]: </span>
<span class="go">0    1</span>
<span class="go">1    2</span>
<span class="go">2    3</span>
<span class="go">3    1</span>
<span class="go">dtype: category</span>

<span class="gp">In [70]: </span><span class="n">s2</span><span class="o">.</span><span class="n">sort</span><span class="p">()</span>

<span class="gp">In [71]: </span><span class="n">s2</span>
<span class="gr">Out[71]: </span>
<span class="go">1    2</span>
<span class="go">2    3</span>
<span class="go">0    1</span>
<span class="go">3    1</span>
<span class="go">dtype: category</span>

<span class="gp">In [72]: </span><span class="k">print</span><span class="p">(</span><span class="n">s2</span><span class="o">.</span><span class="n">min</span><span class="p">(),</span> <span class="n">s2</span><span class="o">.</span><span class="n">max</span><span class="p">())</span>
<span class="go">(2, 1)</span>
</pre></div>
</div>

<div class="admonition note">
<p class="first admonition-title">Note</p>
<p class="last">Note the difference between assigning new level names and reordering the levels: the first
renames levels and therefore the individual values in the <cite>Series</cite>, but if the first
position was sorted last, the renamed value will still be sorted last. Reordering means that the
way values are sorted is different afterwards, but not that individual values in the
<cite>Series</cite> are changed.</p>
</div>

</div>

<div class="section" id="operations">
<h2>Operations<a class="headerlink" href="#operations" title="Permalink to this headline">¶</a></h2>
<p>The following operations are possible with categorical data:</p>
<p>Getting the minimum and maximum, if the categorical is ordered:</p>
<div class="highlight-ipython"><div class="highlight"><pre><span class="gp">In [73]: </span><span class="n">s</span> <span class="o">=</span> <span class="n">pd</span><span class="o">.</span><span class="n">Series</span><span class="p">(</span><span class="n">pd</span><span class="o">.</span><span class="n">Categorical</span><span class="p">([</span><span class="s">&quot;a&quot;</span><span class="p">,</span><span class="s">&quot;b&quot;</span><span class="p">,</span><span class="s">&quot;c&quot;</span><span class="p">,</span><span class="s">&quot;a&quot;</span><span class="p">],</span> <span class="n">levels</span><span class="o">=</span><span class="p">[</span><span class="s">&quot;c&quot;</span><span class="p">,</span><span class="s">&quot;a&quot;</span><span class="p">,</span><span class="s">&quot;b&quot;</span><span class="p">,</span><span class="s">&quot;d&quot;</span><span class="p">]))</span>

<span class="gp">In [74]: </span><span class="k">print</span><span class="p">(</span><span class="n">s</span><span class="o">.</span><span class="n">min</span><span class="p">(),</span> <span class="n">s</span><span class="o">.</span><span class="n">max</span><span class="p">())</span>
<span class="go">(&#39;c&#39;, &#39;b&#39;)</span>
</pre></div>
</div>

<div class="admonition note">
<p class="first admonition-title">Note</p>
<p class="last">If the <cite>Categorical</cite> is not ordered, <tt class="docutils literal"><span class="pre">Categorical.min()</span></tt> and <tt class="docutils literal"><span class="pre">Categorical.max()</span></tt> and the
corresponding operations on <cite>Series</cite> will raise <cite>TypeError</cite>.</p>
</div>

<p>The mode:</p>

<div class="highlight-ipython"><div class="highlight"><pre><span class="gp">In [75]: </span><span class="n">raw_cat</span> <span class="o">=</span> <span class="n">pd</span><span class="o">.</span><span class="n">Categorical</span><span class="p">([</span><span class="s">&quot;a&quot;</span><span class="p">,</span><span class="s">&quot;b&quot;</span><span class="p">,</span><span class="s">&quot;c&quot;</span><span class="p">,</span><span class="s">&quot;c&quot;</span><span class="p">],</span> <span class="n">levels</span><span class="o">=</span><span class="p">[</span><span class="s">&quot;c&quot;</span><span class="p">,</span><span class="s">&quot;a&quot;</span><span class="p">,</span><span class="s">&quot;b&quot;</span><span class="p">,</span><span class="s">&quot;d&quot;</span><span class="p">])</span>

<span class="gp">In [76]: </span><span class="n">s</span> <span class="o">=</span> <span class="n">pd</span><span class="o">.</span><span class="n">Series</span><span class="p">(</span><span class="n">raw_cat</span><span class="p">)</span>

<span class="gp">In [77]: </span><span class="n">raw_cat</span><span class="o">.</span><span class="n">mode</span><span class="p">()</span>
<span class="gr">Out[77]: </span>
<span class="go"> c</span>
<span class="go">Levels (4): Index([&#39;c&#39;, &#39;a&#39;, &#39;b&#39;, &#39;d&#39;], dtype=object), ordered</span>

<span class="gp">In [78]: </span><span class="n">s</span><span class="o">.</span><span class="n">mode</span><span class="p">()</span>
<span class="gr">Out[78]: </span>
<span class="go">0    c</span>
<span class="go">dtype: category</span>
</pre></div>
</div>

<div class="admonition note">
<p class="first admonition-title">Note</p>
<p class="last">Numeric operations like <tt class="docutils literal"><span class="pre">+</span></tt>, <tt class="docutils literal"><span class="pre">-</span></tt>, <tt class="docutils literal"><span class="pre">*</span></tt>, <tt class="docutils literal"><span class="pre">/</span></tt> and operations based on them (e.g.
<tt class="docutils literal"><span class="pre">.median()</span></tt>, which would need to compute the mean between two values if the length of an
array is even) do not work and raise a <cite>TypeError</cite>.</p>
</div>

<p><cite>Series</cite> methods like <cite>Series.value_counts()</cite> will use all levels, even if some levels are not
present in the data:</p>

<div class="highlight-ipython"><div class="highlight"><pre><span class="gp">In [79]: </span><span class="n">s</span> <span class="o">=</span> <span class="n">pd</span><span class="o">.</span><span class="n">Series</span><span class="p">(</span><span class="n">pd</span><span class="o">.</span><span class="n">Categorical</span><span class="p">([</span><span class="s">&quot;a&quot;</span><span class="p">,</span><span class="s">&quot;b&quot;</span><span class="p">,</span><span class="s">&quot;c&quot;</span><span class="p">,</span><span class="s">&quot;c&quot;</span><span class="p">],</span> <span class="n">levels</span><span class="o">=</span><span class="p">[</span><span class="s">&quot;c&quot;</span><span class="p">,</span><span class="s">&quot;a&quot;</span><span class="p">,</span><span class="s">&quot;b&quot;</span><span class="p">,</span><span class="s">&quot;d&quot;</span><span class="p">]))</span>

<span class="gp">In [80]: </span><span class="n">s</span><span class="o">.</span><span class="n">value_counts</span><span class="p">()</span>
<span class="gr">Out[80]: </span>
<span class="go">c    2</span>
<span class="go">b    1</span>
<span class="go">a    1</span>
<span class="go">d    0</span>
<span class="go">dtype: int64</span>
</pre></div>
</div>

<p>Groupby will also show &#8220;unused&#8221; levels:</p>

<div class="highlight-ipython"><div class="highlight"><pre><span class="gp">In [81]: </span><span class="n">cats</span> <span class="o">=</span> <span class="n">pd</span><span class="o">.</span><span class="n">Categorical</span><span class="p">([</span><span class="s">&quot;a&quot;</span><span class="p">,</span><span class="s">&quot;b&quot;</span><span class="p">,</span><span class="s">&quot;b&quot;</span><span class="p">,</span><span class="s">&quot;b&quot;</span><span class="p">,</span><span class="s">&quot;c&quot;</span><span class="p">,</span><span class="s">&quot;c&quot;</span><span class="p">,</span><span class="s">&quot;c&quot;</span><span class="p">],</span> <span class="n">levels</span><span class="o">=</span><span class="p">[</span><span class="s">&quot;a&quot;</span><span class="p">,</span><span class="s">&quot;b&quot;</span><span class="p">,</span><span class="s">&quot;c&quot;</span><span class="p">,</span><span class="s">&quot;d&quot;</span><span class="p">])</span>

<span class="gp">In [82]: </span><span class="n">df</span> <span class="o">=</span> <span class="n">pd</span><span class="o">.</span><span class="n">DataFrame</span><span class="p">({</span><span class="s">&quot;cats&quot;</span><span class="p">:</span><span class="n">cats</span><span class="p">,</span><span class="s">&quot;values&quot;</span><span class="p">:[</span><span class="mi">1</span><span class="p">,</span><span class="mi">2</span><span class="p">,</span><span class="mi">2</span><span class="p">,</span><span class="mi">2</span><span class="p">,</span><span class="mi">3</span><span class="p">,</span><span class="mi">4</span><span class="p">,</span><span class="mi">5</span><span class="p">]})</span>

<span class="gp">In [83]: </span><span class="n">df</span><span class="o">.</span><span class="n">groupby</span><span class="p">(</span><span class="s">&quot;cats&quot;</span><span class="p">)</span><span class="o">.</span><span class="n">mean</span><span class="p">()</span>
<span class="gr">Out[83]: </span>
<span class="go">      values</span>
<span class="go">cats        </span>
<span class="go">a          1</span>
<span class="go">b          2</span>
<span class="go">c          4</span>
<span class="go">d        NaN</span>

<span class="gp">In [84]: </span><span class="n">cats2</span> <span class="o">=</span> <span class="n">pd</span><span class="o">.</span><span class="n">Categorical</span><span class="p">([</span><span class="s">&quot;a&quot;</span><span class="p">,</span><span class="s">&quot;a&quot;</span><span class="p">,</span><span class="s">&quot;b&quot;</span><span class="p">,</span><span class="s">&quot;b&quot;</span><span class="p">],</span> <span class="n">levels</span><span class="o">=</span><span class="p">[</span><span class="s">&quot;a&quot;</span><span class="p">,</span><span class="s">&quot;b&quot;</span><span class="p">,</span><span class="s">&quot;c&quot;</span><span class="p">])</span>

<span class="gp">In [85]: </span><span class="n">df2</span> <span class="o">=</span> <span class="n">pd</span><span class="o">.</span><span class="n">DataFrame</span><span class="p">({</span><span class="s">&quot;cats&quot;</span><span class="p">:</span><span class="n">cats2</span><span class="p">,</span><span class="s">&quot;B&quot;</span><span class="p">:[</span><span class="s">&quot;c&quot;</span><span class="p">,</span><span class="s">&quot;d&quot;</span><span class="p">,</span><span class="s">&quot;c&quot;</span><span class="p">,</span><span class="s">&quot;d&quot;</span><span class="p">],</span> <span class="s">&quot;values&quot;</span><span class="p">:[</span><span class="mi">1</span><span class="p">,</span><span class="mi">2</span><span class="p">,</span><span class="mi">3</span><span class="p">,</span><span class="mi">4</span><span class="p">]})</span>

<span class="c"># This doesn&#39;t work yet with two columns -&gt; see failing unittests</span>
<span class="gp">In [86]: </span><span class="n">df2</span><span class="o">.</span><span class="n">groupby</span><span class="p">([</span><span class="s">&quot;cats&quot;</span><span class="p">,</span><span class="s">&quot;B&quot;</span><span class="p">])</span><span class="o">.</span><span class="n">mean</span><span class="p">()</span>
<span class="gr">Out[86]: </span>
<span class="go">        values</span>
<span class="go">cats B        </span>
<span class="go">a    c       1</span>
<span class="go">     d       2</span>
<span class="go">b    c       3</span>
<span class="go">     d       4</span>
</pre></div>
</div>

<p>Pivot tables:</p>

<div class="highlight-ipython"><div class="highlight"><pre><span class="gp">In [87]: </span><span class="n">raw_cat</span> <span class="o">=</span> <span class="n">pd</span><span class="o">.</span><span class="n">Categorical</span><span class="p">([</span><span class="s">&quot;a&quot;</span><span class="p">,</span><span class="s">&quot;a&quot;</span><span class="p">,</span><span class="s">&quot;b&quot;</span><span class="p">,</span><span class="s">&quot;b&quot;</span><span class="p">],</span> <span class="n">levels</span><span class="o">=</span><span class="p">[</span><span class="s">&quot;a&quot;</span><span class="p">,</span><span class="s">&quot;b&quot;</span><span class="p">,</span><span class="s">&quot;c&quot;</span><span class="p">])</span>

<span class="gp">In [88]: </span><span class="n">df</span> <span class="o">=</span> <span class="n">pd</span><span class="o">.</span><span class="n">DataFrame</span><span class="p">({</span><span class="s">&quot;A&quot;</span><span class="p">:</span><span class="n">raw_cat</span><span class="p">,</span><span class="s">&quot;B&quot;</span><span class="p">:[</span><span class="s">&quot;c&quot;</span><span class="p">,</span><span class="s">&quot;d&quot;</span><span class="p">,</span><span class="s">&quot;c&quot;</span><span class="p">,</span><span class="s">&quot;d&quot;</span><span class="p">],</span> <span class="s">&quot;values&quot;</span><span class="p">:[</span><span class="mi">1</span><span class="p">,</span><span class="mi">2</span><span class="p">,</span><span class="mi">3</span><span class="p">,</span><span class="mi">4</span><span class="p">]})</span>

<span class="gp">In [89]: </span><span class="n">pd</span><span class="o">.</span><span class="n">pivot_table</span><span class="p">(</span><span class="n">df</span><span class="p">,</span> <span class="n">values</span><span class="o">=</span><span class="s">&#39;values&#39;</span><span class="p">,</span> <span class="n">index</span><span class="o">=</span><span class="p">[</span><span class="s">&#39;A&#39;</span><span class="p">,</span> <span class="s">&#39;B&#39;</span><span class="p">])</span>
<span class="gr">Out[89]: </span>
<span class="go">A  B</span>
<span class="go">a  c    1</span>
<span class="go">   d    2</span>
<span class="go">b  c    3</span>
<span class="go">   d    4</span>
<span class="go">Name: values, dtype: int64</span>
</pre></div>
</div>

</div>

<div class="section" id="data-munging">
<h2>Data munging<a class="headerlink" href="#data-munging" title="Permalink to this headline">¶</a></h2>
<p>The optimized pandas data access methods  <tt class="docutils literal"><span class="pre">.loc</span></tt>, <tt class="docutils literal"><span class="pre">.iloc</span></tt>, <tt class="docutils literal"><span class="pre">.ix</span></tt> <tt class="docutils literal"><span class="pre">.at</span></tt>, and <tt class="docutils literal"><span class="pre">.iat</span></tt>,
work as normal, the only difference is the return type (for getting) and
that only values already in the levels can be assigned.</p>
<div class="section" id="getting">
<h3>Getting<a class="headerlink" href="#getting" title="Permalink to this headline">¶</a></h3>
<p>If the slicing operation returns either a <cite>DataFrame</cite> or a a column of type <cite>Series</cite>,
the <tt class="docutils literal"><span class="pre">category</span></tt> dtype is preserved.</p>
<div class="highlight-ipython"><div class="highlight"><pre><span class="gp">In [90]: </span><span class="n">cats</span> <span class="o">=</span> <span class="n">pd</span><span class="o">.</span><span class="n">Categorical</span><span class="p">([</span><span class="s">&quot;a&quot;</span><span class="p">,</span><span class="s">&quot;b&quot;</span><span class="p">,</span><span class="s">&quot;b&quot;</span><span class="p">,</span><span class="s">&quot;b&quot;</span><span class="p">,</span><span class="s">&quot;c&quot;</span><span class="p">,</span><span class="s">&quot;c&quot;</span><span class="p">,</span><span class="s">&quot;c&quot;</span><span class="p">],</span> <span class="n">levels</span><span class="o">=</span><span class="p">[</span><span class="s">&quot;a&quot;</span><span class="p">,</span><span class="s">&quot;b&quot;</span><span class="p">,</span><span class="s">&quot;c&quot;</span><span class="p">])</span>

<span class="gp">In [91]: </span><span class="n">idx</span> <span class="o">=</span> <span class="n">pd</span><span class="o">.</span><span class="n">Index</span><span class="p">([</span><span class="s">&quot;h&quot;</span><span class="p">,</span><span class="s">&quot;i&quot;</span><span class="p">,</span><span class="s">&quot;j&quot;</span><span class="p">,</span><span class="s">&quot;k&quot;</span><span class="p">,</span><span class="s">&quot;l&quot;</span><span class="p">,</span><span class="s">&quot;m&quot;</span><span class="p">,</span><span class="s">&quot;n&quot;</span><span class="p">,])</span>

<span class="gp">In [92]: </span><span class="n">values</span><span class="o">=</span> <span class="p">[</span><span class="mi">1</span><span class="p">,</span><span class="mi">2</span><span class="p">,</span><span class="mi">2</span><span class="p">,</span><span class="mi">2</span><span class="p">,</span><span class="mi">3</span><span class="p">,</span><span class="mi">4</span><span class="p">,</span><span class="mi">5</span><span class="p">]</span>

<span class="gp">In [93]: </span><span class="n">df</span> <span class="o">=</span> <span class="n">pd</span><span class="o">.</span><span class="n">DataFrame</span><span class="p">({</span><span class="s">&quot;cats&quot;</span><span class="p">:</span><span class="n">cats</span><span class="p">,</span><span class="s">&quot;values&quot;</span><span class="p">:</span><span class="n">values</span><span class="p">},</span> <span class="n">index</span><span class="o">=</span><span class="n">idx</span><span class="p">)</span>

<span class="gp">In [94]: </span><span class="n">df</span><span class="o">.</span><span class="n">iloc</span><span class="p">[</span><span class="mi">2</span><span class="p">:</span><span class="mi">4</span><span class="p">,:]</span>
<span class="gr">Out[94]: </span>
<span class="go">  cats  values</span>
<span class="go">j    b       2</span>
<span class="go">k    b       2</span>

<span class="gp">In [95]: </span><span class="n">df</span><span class="o">.</span><span class="n">iloc</span><span class="p">[</span><span class="mi">2</span><span class="p">:</span><span class="mi">4</span><span class="p">,:]</span><span class="o">.</span><span class="n">dtypes</span>
<span class="gr">Out[95]: </span>
<span class="go">cats      category</span>
<span class="go">values       int64</span>
<span class="go">dtype: object</span>

<span class="gp">In [96]: </span><span class="n">df</span><span class="o">.</span><span class="n">loc</span><span class="p">[</span><span class="s">&quot;h&quot;</span><span class="p">:</span><span class="s">&quot;j&quot;</span><span class="p">,</span><span class="s">&quot;cats&quot;</span><span class="p">]</span>
<span class="gr">Out[96]: </span>
<span class="go">h    a</span>
<span class="go">i    b</span>
<span class="go">j    b</span>
<span class="go">Name: cats, dtype: category</span>

<span class="gp">In [97]: </span><span class="n">df</span><span class="o">.</span><span class="n">ix</span><span class="p">[</span><span class="s">&quot;h&quot;</span><span class="p">:</span><span class="s">&quot;j&quot;</span><span class="p">,</span><span class="mi">0</span><span class="p">:</span><span class="mi">1</span><span class="p">]</span>
<span class="gr">Out[97]: </span>
<span class="go">  cats</span>
<span class="go">h    a</span>
<span class="go">i    b</span>
<span class="go">j    b</span>

<span class="gp">In [98]: </span><span class="n">df</span><span class="p">[</span><span class="n">df</span><span class="p">[</span><span class="s">&quot;cats&quot;</span><span class="p">]</span> <span class="o">==</span> <span class="s">&quot;b&quot;</span><span class="p">]</span>
<span class="gr">Out[98]: </span>
<span class="go">  cats  values</span>
<span class="go">i    b       2</span>
<span class="go">j    b       2</span>
<span class="go">k    b       2</span>
</pre></div>
</div>

<p>An example where the <cite>Categorical</cite> is not preserved is if you take one single row: the
resulting <cite>Series</cite> is of dtype <tt class="docutils literal"><span class="pre">object</span></tt>:</p>

<div class="highlight-ipython"><div class="highlight"><pre><span class="c"># get the complete &quot;h&quot; row as a Series</span>
<span class="gp">In [99]: </span><span class="n">df</span><span class="o">.</span><span class="n">loc</span><span class="p">[</span><span class="s">&quot;h&quot;</span><span class="p">,</span> <span class="p">:]</span>
<span class="gr">Out[99]: </span>
<span class="go">cats      a</span>
<span class="go">values    1</span>
<span class="go">Name: h, dtype: object</span>
</pre></div>
</div>

<p>Returning a single item from a <cite>Categorical</cite> will also return the value, not a <cite>Categorical</cite>
of length &#8220;1&#8221;.</p>

<div class="highlight-ipython"><div class="highlight"><pre><span class="gp">In [100]: </span><span class="n">df</span><span class="o">.</span><span class="n">iat</span><span class="p">[</span><span class="mi">0</span><span class="p">,</span><span class="mi">0</span><span class="p">]</span>
<span class="gr">Out[100]: </span><span class="s">&#39;a&#39;</span>

<span class="gp">In [101]: </span><span class="n">df</span><span class="p">[</span><span class="s">&quot;cats&quot;</span><span class="p">]</span><span class="o">.</span><span class="n">cat</span><span class="o">.</span><span class="n">levels</span> <span class="o">=</span> <span class="p">[</span><span class="s">&quot;x&quot;</span><span class="p">,</span><span class="s">&quot;y&quot;</span><span class="p">,</span><span class="s">&quot;z&quot;</span><span class="p">]</span>

<span class="gp">In [102]: </span><span class="n">df</span><span class="o">.</span><span class="n">at</span><span class="p">[</span><span class="s">&quot;h&quot;</span><span class="p">,</span><span class="s">&quot;cats&quot;</span><span class="p">]</span> <span class="c"># returns a string</span>
<span class="gr">Out[102]: </span><span class="s">&#39;x&#39;</span>
</pre></div>
</div>

<div class="admonition note">
<p class="first admonition-title">Note</p>
<p class="last">This is a difference to R&#8217;s <cite>factor</cite> function, where <tt class="docutils literal"><span class="pre">factor(c(1,2,3))[1]</span></tt>
returns a single value <cite>factor</cite>.</p>
</div>

<p>To get a single value <cite>Series</cite> of type <tt class="docutils literal"><span class="pre">category</span></tt> pass in a single value list:</p>

<div class="highlight-ipython"><div class="highlight"><pre><span class="gp">In [103]: </span><span class="n">df</span><span class="o">.</span><span class="n">loc</span><span class="p">[[</span><span class="s">&quot;h&quot;</span><span class="p">],</span><span class="s">&quot;cats&quot;</span><span class="p">]</span>
<span class="gr">Out[103]: </span>
<span class="go">h    x</span>
<span class="go">Name: cats, dtype: category</span>
</pre></div>
</div>

</div>

<div class="section" id="setting">
<h3>Setting<a class="headerlink" href="#setting" title="Permalink to this headline">¶</a></h3>
<p>Setting values in a categorical column (or <cite>Series</cite>) works as long as the value is included in the
<cite>levels</cite>:</p>
<div class="highlight-ipython"><div class="highlight"><pre><span class="gp">In [104]: </span><span class="n">cats</span> <span class="o">=</span> <span class="n">pd</span><span class="o">.</span><span class="n">Categorical</span><span class="p">([</span><span class="s">&quot;a&quot;</span><span class="p">,</span><span class="s">&quot;a&quot;</span><span class="p">,</span><span class="s">&quot;a&quot;</span><span class="p">,</span><span class="s">&quot;a&quot;</span><span class="p">,</span><span class="s">&quot;a&quot;</span><span class="p">,</span><span class="s">&quot;a&quot;</span><span class="p">,</span><span class="s">&quot;a&quot;</span><span class="p">],</span> <span class="n">levels</span><span class="o">=</span><span class="p">[</span><span class="s">&quot;a&quot;</span><span class="p">,</span><span class="s">&quot;b&quot;</span><span class="p">])</span>

<span class="gp">In [105]: </span><span class="n">idx</span> <span class="o">=</span> <span class="n">pd</span><span class="o">.</span><span class="n">Index</span><span class="p">([</span><span class="s">&quot;h&quot;</span><span class="p">,</span><span class="s">&quot;i&quot;</span><span class="p">,</span><span class="s">&quot;j&quot;</span><span class="p">,</span><span class="s">&quot;k&quot;</span><span class="p">,</span><span class="s">&quot;l&quot;</span><span class="p">,</span><span class="s">&quot;m&quot;</span><span class="p">,</span><span class="s">&quot;n&quot;</span><span class="p">])</span>

<span class="gp">In [106]: </span><span class="n">values</span> <span class="o">=</span> <span class="p">[</span><span class="mi">1</span><span class="p">,</span><span class="mi">1</span><span class="p">,</span><span class="mi">1</span><span class="p">,</span><span class="mi">1</span><span class="p">,</span><span class="mi">1</span><span class="p">,</span><span class="mi">1</span><span class="p">,</span><span class="mi">1</span><span class="p">]</span>

<span class="gp">In [107]: </span><span class="n">df</span> <span class="o">=</span> <span class="n">pd</span><span class="o">.</span><span class="n">DataFrame</span><span class="p">({</span><span class="s">&quot;cats&quot;</span><span class="p">:</span><span class="n">cats</span><span class="p">,</span><span class="s">&quot;values&quot;</span><span class="p">:</span><span class="n">values</span><span class="p">},</span> <span class="n">index</span><span class="o">=</span><span class="n">idx</span><span class="p">)</span>

<span class="gp">In [108]: </span><span class="n">df</span><span class="o">.</span><span class="n">iloc</span><span class="p">[</span><span class="mi">2</span><span class="p">:</span><span class="mi">4</span><span class="p">,:]</span> <span class="o">=</span> <span class="p">[[</span><span class="s">&quot;b&quot;</span><span class="p">,</span><span class="mi">2</span><span class="p">],[</span><span class="s">&quot;b&quot;</span><span class="p">,</span><span class="mi">2</span><span class="p">]]</span>

<span class="gp">In [109]: </span><span class="n">df</span>
<span class="gr">Out[109]: </span>
<span class="go">  cats  values</span>
<span class="go">h    a       1</span>
<span class="go">i    a       1</span>
<span class="go">j    b       2</span>
<span class="go">k    b       2</span>
<span class="go">l    a       1</span>
<span class="go">m    a       1</span>
<span class="go">n    a       1</span>

<span class="gp">In [110]: </span><span class="k">try</span><span class="p">:</span>
<span class="gp">   .....:</span>     <span class="n">df</span><span class="o">.</span><span class="n">iloc</span><span class="p">[</span><span class="mi">2</span><span class="p">:</span><span class="mi">4</span><span class="p">,:]</span> <span class="o">=</span> <span class="p">[[</span><span class="s">&quot;c&quot;</span><span class="p">,</span><span class="mi">3</span><span class="p">],[</span><span class="s">&quot;c&quot;</span><span class="p">,</span><span class="mi">3</span><span class="p">]]</span>
<span class="gp">   .....:</span> <span class="k">except</span> <span class="ne">ValueError</span> <span class="k">as</span> <span class="n">e</span><span class="p">:</span>
<span class="gp">   .....:</span>     <span class="k">print</span><span class="p">(</span><span class="s">&quot;ValueError: &quot;</span> <span class="o">+</span> <span class="nb">str</span><span class="p">(</span><span class="n">e</span><span class="p">))</span>
<span class="gp">   .....:</span> 
<span class="go">ValueError: cannot setitem on a Categorical with a new level, set the levels first</span>
</pre></div>
</div>

<p>Setting values by assigning a <cite>Categorical</cite> will also check that the <cite>levels</cite> match:</p>

<div class="highlight-ipython"><div class="highlight"><pre><span class="gp">In [111]: </span><span class="n">df</span><span class="o">.</span><span class="n">loc</span><span class="p">[</span><span class="s">&quot;j&quot;</span><span class="p">:</span><span class="s">&quot;k&quot;</span><span class="p">,</span><span class="s">&quot;cats&quot;</span><span class="p">]</span> <span class="o">=</span> <span class="n">pd</span><span class="o">.</span><span class="n">Categorical</span><span class="p">([</span><span class="s">&quot;a&quot;</span><span class="p">,</span><span class="s">&quot;a&quot;</span><span class="p">],</span> <span class="n">levels</span><span class="o">=</span><span class="p">[</span><span class="s">&quot;a&quot;</span><span class="p">,</span><span class="s">&quot;b&quot;</span><span class="p">])</span>

<span class="gp">In [112]: </span><span class="n">df</span>
<span class="gr">Out[112]: </span>
<span class="go">  cats  values</span>
<span class="go">h    a       1</span>
<span class="go">i    a       1</span>
<span class="go">j    a       2</span>
<span class="go">k    a       2</span>
<span class="go">l    a       1</span>
<span class="go">m    a       1</span>
<span class="go">n    a       1</span>

<span class="gp">In [113]: </span><span class="k">try</span><span class="p">:</span>
<span class="gp">   .....:</span>     <span class="n">df</span><span class="o">.</span><span class="n">loc</span><span class="p">[</span><span class="s">&quot;j&quot;</span><span class="p">:</span><span class="s">&quot;k&quot;</span><span class="p">,</span><span class="s">&quot;cats&quot;</span><span class="p">]</span> <span class="o">=</span> <span class="n">pd</span><span class="o">.</span><span class="n">Categorical</span><span class="p">([</span><span class="s">&quot;b&quot;</span><span class="p">,</span><span class="s">&quot;b&quot;</span><span class="p">],</span> <span class="n">levels</span><span class="o">=</span><span class="p">[</span><span class="s">&quot;a&quot;</span><span class="p">,</span><span class="s">&quot;b&quot;</span><span class="p">,</span><span class="s">&quot;c&quot;</span><span class="p">])</span>
<span class="gp">   .....:</span> <span class="k">except</span> <span class="ne">ValueError</span> <span class="k">as</span> <span class="n">e</span><span class="p">:</span>
<span class="gp">   .....:</span>     <span class="k">print</span><span class="p">(</span><span class="s">&quot;ValueError: &quot;</span> <span class="o">+</span> <span class="nb">str</span><span class="p">(</span><span class="n">e</span><span class="p">))</span>
<span class="gp">   .....:</span> 
<span class="go">ValueError: cannot set a Categorical with another, without identical levels</span>
</pre></div>
</div>

<p>Assigning a <cite>Categorical</cite> to parts of a column of other types will use the values:</p>

<div class="highlight-ipython"><div class="highlight"><pre><span class="gp">In [114]: </span><span class="n">df</span> <span class="o">=</span> <span class="n">pd</span><span class="o">.</span><span class="n">DataFrame</span><span class="p">({</span><span class="s">&quot;a&quot;</span><span class="p">:[</span><span class="mi">1</span><span class="p">,</span><span class="mi">1</span><span class="p">,</span><span class="mi">1</span><span class="p">,</span><span class="mi">1</span><span class="p">,</span><span class="mi">1</span><span class="p">],</span> <span class="s">&quot;b&quot;</span><span class="p">:[</span><span class="s">&quot;a&quot;</span><span class="p">,</span><span class="s">&quot;a&quot;</span><span class="p">,</span><span class="s">&quot;a&quot;</span><span class="p">,</span><span class="s">&quot;a&quot;</span><span class="p">,</span><span class="s">&quot;a&quot;</span><span class="p">]})</span>

<span class="gp">In [115]: </span><span class="n">df</span><span class="o">.</span><span class="n">loc</span><span class="p">[</span><span class="mi">1</span><span class="p">:</span><span class="mi">2</span><span class="p">,</span><span class="s">&quot;a&quot;</span><span class="p">]</span> <span class="o">=</span> <span class="n">pd</span><span class="o">.</span><span class="n">Categorical</span><span class="p">([</span><span class="s">&quot;b&quot;</span><span class="p">,</span><span class="s">&quot;b&quot;</span><span class="p">],</span> <span class="n">levels</span><span class="o">=</span><span class="p">[</span><span class="s">&quot;a&quot;</span><span class="p">,</span><span class="s">&quot;b&quot;</span><span class="p">])</span>

<span class="gp">In [116]: </span><span class="n">df</span><span class="o">.</span><span class="n">loc</span><span class="p">[</span><span class="mi">2</span><span class="p">:</span><span class="mi">3</span><span class="p">,</span><span class="s">&quot;b&quot;</span><span class="p">]</span> <span class="o">=</span> <span class="n">pd</span><span class="o">.</span><span class="n">Categorical</span><span class="p">([</span><span class="s">&quot;b&quot;</span><span class="p">,</span><span class="s">&quot;b&quot;</span><span class="p">],</span> <span class="n">levels</span><span class="o">=</span><span class="p">[</span><span class="s">&quot;a&quot;</span><span class="p">,</span><span class="s">&quot;b&quot;</span><span class="p">])</span>

<span class="gp">In [117]: </span><span class="n">df</span>
<span class="gr">Out[117]: </span>
<span class="go">   a  b</span>
<span class="go">0  1  a</span>
<span class="go">1  b  a</span>
<span class="go">2  b  b</span>
<span class="go">3  1  b</span>
<span class="go">4  1  a</span>

<span class="gp">In [118]: </span><span class="n">df</span><span class="o">.</span><span class="n">dtypes</span>
<span class="gr">Out[118]: </span>
<span class="go">a    object</span>
<span class="go">b    object</span>
<span class="go">dtype: object</span>
</pre></div>
</div>

</div>

<div class="section" id="merging">
<h3>Merging<a class="headerlink" href="#merging" title="Permalink to this headline">¶</a></h3>
<p>You can concat two <cite>DataFrames</cite> containing categorical data together,
but the levels of these <cite>Categoricals</cite> need to be the same:</p>
<div class="highlight-ipython"><div class="highlight"><pre><span class="gp">In [119]: </span><span class="n">cat</span> <span class="o">=</span> <span class="n">pd</span><span class="o">.</span><span class="n">Categorical</span><span class="p">([</span><span class="s">&quot;a&quot;</span><span class="p">,</span><span class="s">&quot;b&quot;</span><span class="p">],</span> <span class="n">levels</span><span class="o">=</span><span class="p">[</span><span class="s">&quot;a&quot;</span><span class="p">,</span><span class="s">&quot;b&quot;</span><span class="p">])</span>

<span class="gp">In [120]: </span><span class="n">vals</span> <span class="o">=</span> <span class="p">[</span><span class="mi">1</span><span class="p">,</span><span class="mi">2</span><span class="p">]</span>

<span class="gp">In [121]: </span><span class="n">df</span> <span class="o">=</span> <span class="n">pd</span><span class="o">.</span><span class="n">DataFrame</span><span class="p">({</span><span class="s">&quot;cats&quot;</span><span class="p">:</span><span class="n">cat</span><span class="p">,</span> <span class="s">&quot;vals&quot;</span><span class="p">:</span><span class="n">vals</span><span class="p">})</span>

<span class="gp">In [122]: </span><span class="n">res</span> <span class="o">=</span> <span class="n">pd</span><span class="o">.</span><span class="n">concat</span><span class="p">([</span><span class="n">df</span><span class="p">,</span><span class="n">df</span><span class="p">])</span>

<span class="gp">In [123]: </span><span class="n">res</span>
<span class="gr">Out[123]: </span>
<span class="go">  cats  vals</span>
<span class="go">0    a     1</span>
<span class="go">1    b     2</span>
<span class="go">0    a     1</span>
<span class="go">1    b     2</span>

<span class="gp">In [124]: </span><span class="n">res</span><span class="o">.</span><span class="n">dtypes</span>
<span class="gr">Out[124]: </span>
<span class="go">cats    category</span>
<span class="go">vals       int64</span>
<span class="go">dtype: object</span>

<span class="gp">In [125]: </span><span class="n">df_different</span> <span class="o">=</span> <span class="n">df</span><span class="o">.</span><span class="n">copy</span><span class="p">()</span>

<span class="gp">In [126]: </span><span class="n">df_different</span><span class="p">[</span><span class="s">&quot;cats&quot;</span><span class="p">]</span><span class="o">.</span><span class="n">cat</span><span class="o">.</span><span class="n">levels</span> <span class="o">=</span> <span class="p">[</span><span class="s">&quot;a&quot;</span><span class="p">,</span><span class="s">&quot;b&quot;</span><span class="p">,</span><span class="s">&quot;c&quot;</span><span class="p">]</span>

<span class="gp">In [127]: </span><span class="k">try</span><span class="p">:</span>
<span class="gp">   .....:</span>     <span class="n">pd</span><span class="o">.</span><span class="n">concat</span><span class="p">([</span><span class="n">df</span><span class="p">,</span><span class="n">df</span><span class="p">])</span>
<span class="gp">   .....:</span> <span class="k">except</span> <span class="ne">ValueError</span> <span class="k">as</span> <span class="n">e</span><span class="p">:</span>
<span class="gp">   .....:</span>     <span class="k">print</span><span class="p">(</span><span class="s">&quot;ValueError: &quot;</span> <span class="o">+</span> <span class="nb">str</span><span class="p">(</span><span class="n">e</span><span class="p">))</span>
<span class="gp">   .....:</span> 
</pre></div>
</div>

<p>The same applies to <tt class="docutils literal"><span class="pre">df.append(df)</span></tt>.</p>

</div>
</div>

<div class="section" id="getting-data-in-out">
<h2>Getting Data In/Out<a class="headerlink" href="#getting-data-in-out" title="Permalink to this headline">¶</a></h2>
<p>Writing data (<cite>Series</cite>, <cite>Frames</cite>) to a HDF store and reading it in entirety works. Querying the hdf
store does not yet work.</p>
<div class="highlight-ipython"><div class="highlight"><pre><span class="gp">In [128]: </span><span class="n">hdf_file</span> <span class="o">=</span> <span class="s">&quot;test.h5&quot;</span>

<span class="gp">In [129]: </span><span class="n">s</span> <span class="o">=</span> <span class="n">pd</span><span class="o">.</span><span class="n">Series</span><span class="p">(</span><span class="n">pd</span><span class="o">.</span><span class="n">Categorical</span><span class="p">([</span><span class="s">&#39;a&#39;</span><span class="p">,</span> <span class="s">&#39;b&#39;</span><span class="p">,</span> <span class="s">&#39;b&#39;</span><span class="p">,</span> <span class="s">&#39;a&#39;</span><span class="p">,</span> <span class="s">&#39;a&#39;</span><span class="p">,</span> <span class="s">&#39;c&#39;</span><span class="p">],</span> <span class="n">levels</span><span class="o">=</span><span class="p">[</span><span class="s">&#39;a&#39;</span><span class="p">,</span><span class="s">&#39;b&#39;</span><span class="p">,</span><span class="s">&#39;c&#39;</span><span class="p">,</span><span class="s">&#39;d&#39;</span><span class="p">]))</span>

<span class="gp">In [130]: </span><span class="n">df</span> <span class="o">=</span> <span class="n">pd</span><span class="o">.</span><span class="n">DataFrame</span><span class="p">({</span><span class="s">&quot;s&quot;</span><span class="p">:</span><span class="n">s</span><span class="p">,</span> <span class="s">&quot;vals&quot;</span><span class="p">:[</span><span class="mi">1</span><span class="p">,</span><span class="mi">2</span><span class="p">,</span><span class="mi">3</span><span class="p">,</span><span class="mi">4</span><span class="p">,</span><span class="mi">5</span><span class="p">,</span><span class="mi">6</span><span class="p">]})</span>

<span class="gp">In [131]: </span><span class="n">df</span><span class="o">.</span><span class="n">to_hdf</span><span class="p">(</span><span class="n">hdf_file</span><span class="p">,</span> <span class="s">&quot;frame&quot;</span><span class="p">)</span>

<span class="gp">In [132]: </span><span class="n">df2</span> <span class="o">=</span> <span class="n">pd</span><span class="o">.</span><span class="n">read_hdf</span><span class="p">(</span><span class="n">hdf_file</span><span class="p">,</span> <span class="s">&quot;frame&quot;</span><span class="p">)</span>

<span class="gp">In [133]: </span><span class="n">df2</span>
<span class="gr">Out[133]: </span>
<span class="go">   s  vals</span>
<span class="go">0  a     1</span>
<span class="go">1  b     2</span>
<span class="go">2  b     3</span>
<span class="go">3  a     4</span>
<span class="go">4  a     5</span>
<span class="go">5  c     6</span>

<span class="gp">In [134]: </span><span class="k">try</span><span class="p">:</span>
<span class="gp">   .....:</span>     <span class="n">pd</span><span class="o">.</span><span class="n">read_hdf</span><span class="p">(</span><span class="n">hdf_file</span><span class="p">,</span> <span class="s">&quot;frame&quot;</span><span class="p">,</span> <span class="n">where</span> <span class="o">=</span> <span class="p">[</span><span class="s">&#39;index&gt;2&#39;</span><span class="p">])</span>
<span class="gp">   .....:</span> <span class="k">except</span> <span class="ne">TypeError</span> <span class="k">as</span> <span class="n">e</span><span class="p">:</span>
<span class="gp">   .....:</span>     <span class="k">print</span><span class="p">(</span><span class="s">&quot;TypeError: &quot;</span> <span class="o">+</span> <span class="nb">str</span><span class="p">(</span><span class="n">e</span><span class="p">))</span>
<span class="gp">   .....:</span> 
<span class="go">TypeError: cannot pass a where specification when reading from a Fixed format store. this store must be selected in its entirety</span>
</pre></div>
</div>

<p>Writing to a csv file will convert the data, effectively removing any information about the
<cite>Categorical</cite> (levels and ordering). So if you read back the csv file you have to convert the
relevant columns back to <cite>category</cite> and assign the right levels and level ordering.</p>

<div class="highlight-ipython"><div class="highlight"><pre><span class="gp">In [135]: </span><span class="n">s</span> <span class="o">=</span> <span class="n">pd</span><span class="o">.</span><span class="n">Series</span><span class="p">(</span><span class="n">pd</span><span class="o">.</span><span class="n">Categorical</span><span class="p">([</span><span class="s">&#39;a&#39;</span><span class="p">,</span> <span class="s">&#39;b&#39;</span><span class="p">,</span> <span class="s">&#39;b&#39;</span><span class="p">,</span> <span class="s">&#39;a&#39;</span><span class="p">,</span> <span class="s">&#39;a&#39;</span><span class="p">,</span> <span class="s">&#39;d&#39;</span><span class="p">]))</span>

<span class="c"># rename the levels</span>
<span class="gp">In [136]: </span><span class="n">s</span><span class="o">.</span><span class="n">cat</span><span class="o">.</span><span class="n">levels</span> <span class="o">=</span> <span class="p">[</span><span class="s">&quot;very good&quot;</span><span class="p">,</span> <span class="s">&quot;good&quot;</span><span class="p">,</span> <span class="s">&quot;bad&quot;</span><span class="p">]</span>

<span class="c"># add new levels at the end</span>
<span class="gp">In [137]: </span><span class="n">s</span><span class="o">.</span><span class="n">cat</span><span class="o">.</span><span class="n">levels</span> <span class="o">=</span> <span class="nb">list</span><span class="p">(</span><span class="n">s</span><span class="o">.</span><span class="n">cat</span><span class="o">.</span><span class="n">levels</span><span class="p">)</span> <span class="o">+</span> <span class="p">[</span><span class="s">&quot;medium&quot;</span><span class="p">,</span> <span class="s">&quot;very bad&quot;</span><span class="p">]</span>

<span class="c"># reorder the levels</span>
<span class="gp">In [138]: </span><span class="n">s</span><span class="o">.</span><span class="n">cat</span><span class="o">.</span><span class="n">reorder_levels</span><span class="p">([</span><span class="s">&quot;very bad&quot;</span><span class="p">,</span> <span class="s">&quot;bad&quot;</span><span class="p">,</span> <span class="s">&quot;medium&quot;</span><span class="p">,</span> <span class="s">&quot;good&quot;</span><span class="p">,</span> <span class="s">&quot;very good&quot;</span><span class="p">])</span>

<span class="gp">In [139]: </span><span class="n">df</span> <span class="o">=</span> <span class="n">pd</span><span class="o">.</span><span class="n">DataFrame</span><span class="p">({</span><span class="s">&quot;s&quot;</span><span class="p">:</span><span class="n">s</span><span class="p">,</span> <span class="s">&quot;vals&quot;</span><span class="p">:[</span><span class="mi">1</span><span class="p">,</span><span class="mi">2</span><span class="p">,</span><span class="mi">3</span><span class="p">,</span><span class="mi">4</span><span class="p">,</span><span class="mi">5</span><span class="p">,</span><span class="mi">6</span><span class="p">]})</span>

<span class="gp">In [140]: </span><span class="n">df</span><span class="o">.</span><span class="n">to_csv</span><span class="p">(</span><span class="n">csv_file</span><span class="p">)</span>
<span class="go">---------------------------------------------------------------------------</span>
<span class="go">IndexError                                Traceback (most recent call last)</span>
<span class="go">&lt;ipython-input-140-72bb1b843e60&gt; in &lt;module&gt;()</span>
<span class="go">----&gt; 1 df.to_csv(csv_file)</span>

<span class="go">c:\data\external\pandas\pandas\util\decorators.pyc in wrapper(*args, **kwargs)</span>
<span class="go">     58                 else:</span>
<span class="go">     59                     kwargs[new_arg_name] = old_arg_value</span>
<span class="go">---&gt; 60             return func(*args, **kwargs)</span>
<span class="go">     61         return wrapper</span>
<span class="go">     62     return _deprecate_kwarg</span>

<span class="go">c:\data\external\pandas\pandas\core\frame.pyc in to_csv(self, path_or_buf, sep, na_rep, float_format, columns, header, index, index_label, mode, encoding, quoting, quotechar, line_terminator, chunksize, tupleize_cols, date_format, doublequote, escapechar, **kwds)</span>
<span class="go">   1139                                      doublequote=doublequote,</span>
<span class="go">   1140                                      escapechar=escapechar)</span>
<span class="go">-&gt; 1141         formatter.save()</span>
<span class="go">   1142 </span>
<span class="go">   1143         if path_or_buf is None:</span>

<span class="go">c:\data\external\pandas\pandas\core\format.pyc in save(self)</span>
<span class="go">   1312 </span>
<span class="go">   1313             else:</span>
<span class="go">-&gt; 1314                 self._save()</span>
<span class="go">   1315 </span>
<span class="go">   1316         finally:</span>

<span class="go">c:\data\external\pandas\pandas\core\format.pyc in _save(self)</span>
<span class="go">   1412                 break</span>
<span class="go">   1413 </span>
<span class="go">-&gt; 1414             self._save_chunk(start_i, end_i)</span>
<span class="go">   1415 </span>
<span class="go">   1416     def _save_chunk(self, start_i, end_i):</span>

<span class="go">c:\data\external\pandas\pandas\core\format.pyc in _save_chunk(self, start_i, end_i)</span>
<span class="go">   1424             d = b.to_native_types(slicer=slicer, na_rep=self.na_rep,</span>
<span class="go">   1425                                   float_format=self.float_format,</span>
<span class="go">-&gt; 1426                                   date_format=self.date_format)</span>
<span class="go">   1427 </span>
<span class="go">   1428             for col_loc, col in zip(b.mgr_locs, d):</span>

<span class="go">c:\data\external\pandas\pandas\core\internals.pyc in to_native_types(self, slicer, na_rep, **kwargs)</span>
<span class="go">    446         values = self.values</span>
<span class="go">    447         if slicer is not None:</span>
<span class="go">--&gt; 448             values = values[:, slicer]</span>
<span class="go">    449         values = np.array(values, dtype=object)</span>
<span class="go">    450         mask = isnull(values)</span>

<span class="go">c:\data\external\pandas\pandas\core\categorical.pyc in __getitem__(self, key)</span>
<span class="go">    669                 return self.levels[i]</span>
<span class="go">    670         else:</span>
<span class="go">--&gt; 671             return Categorical(values=self._codes[key], levels=self.levels,</span>
<span class="go">    672                                ordered=self.ordered, fastpath=True)</span>
<span class="go">    673 </span>

<span class="go">IndexError: too many indices</span>

<span class="gp">In [141]: </span><span class="n">df2</span> <span class="o">=</span> <span class="n">pd</span><span class="o">.</span><span class="n">read_csv</span><span class="p">(</span><span class="n">csv_file</span><span class="p">)</span>
<span class="go">---------------------------------------------------------------------------</span>
<span class="go">CParserError                              Traceback (most recent call last)</span>
<span class="go">&lt;ipython-input-141-8d612f40488f&gt; in &lt;module&gt;()</span>
<span class="go">----&gt; 1 df2 = pd.read_csv(csv_file)</span>

<span class="go">c:\data\external\pandas\pandas\io\parsers.pyc in parser_f(filepath_or_buffer, sep, dialect, compression, doublequote, escapechar, quotechar, quoting, skipinitialspace, lineterminator, header, index_col, names, prefix, skiprows, skipfooter, skip_footer, na_values, na_fvalues, true_values, false_values, delimiter, converters, dtype, usecols, engine, delim_whitespace, as_recarray, na_filter, compact_ints, use_unsigned, low_memory, buffer_lines, warn_bad_lines, error_bad_lines, keep_default_na, thousands, comment, decimal, parse_dates, keep_date_col, dayfirst, date_parser, memory_map, nrows, iterator, chunksize, verbose, encoding, squeeze, mangle_dupe_cols, tupleize_cols, infer_datetime_format)</span>
<span class="go">    450                     infer_datetime_format=infer_datetime_format)</span>
<span class="go">    451 </span>
<span class="go">--&gt; 452         return _read(filepath_or_buffer, kwds)</span>
<span class="go">    453 </span>
<span class="go">    454     parser_f.__name__ = name</span>

<span class="go">c:\data\external\pandas\pandas\io\parsers.pyc in _read(filepath_or_buffer, kwds)</span>
<span class="go">    232 </span>
<span class="go">    233     # Create the parser.</span>
<span class="go">--&gt; 234     parser = TextFileReader(filepath_or_buffer, **kwds)</span>
<span class="go">    235 </span>
<span class="go">    236     if (nrows is not None) and (chunksize is not None):</span>

<span class="go">c:\data\external\pandas\pandas\io\parsers.pyc in __init__(self, f, engine, **kwds)</span>
<span class="go">    540             self.options[&#39;has_index_names&#39;] = kwds[&#39;has_index_names&#39;]</span>
<span class="go">    541 </span>
<span class="go">--&gt; 542         self._make_engine(self.engine)</span>
<span class="go">    543 </span>
<span class="go">    544     def _get_options_with_defaults(self, engine):</span>

<span class="go">c:\data\external\pandas\pandas\io\parsers.pyc in _make_engine(self, engine)</span>
<span class="go">    677     def _make_engine(self, engine=&#39;c&#39;):</span>
<span class="go">    678         if engine == &#39;c&#39;:</span>
<span class="go">--&gt; 679             self._engine = CParserWrapper(self.f, **self.options)</span>
<span class="go">    680         else:</span>
<span class="go">    681             if engine == &#39;python&#39;:</span>

<span class="go">c:\data\external\pandas\pandas\io\parsers.pyc in __init__(self, src, **kwds)</span>
<span class="go">   1039         kwds[&#39;allow_leading_cols&#39;] = self.index_col is not False</span>
<span class="go">   1040 </span>
<span class="go">-&gt; 1041         self._reader = _parser.TextReader(src, **kwds)</span>
<span class="go">   1042 </span>
<span class="go">   1043         # XXX</span>

<span class="go">c:\data\external\pandas\pandas\parser.pyd in pandas.parser.TextReader.__cinit__ (pandas\parser.c:4629)()</span>

<span class="go">c:\data\external\pandas\pandas\parser.pyd in pandas.parser.TextReader._get_header (pandas\parser.c:6092)()</span>

<span class="go">CParserError: Passed header=0 but only 0 lines in file</span>

<span class="gp">In [142]: </span><span class="n">df2</span><span class="o">.</span><span class="n">dtypes</span>
<span class="gr">Out[142]: </span>
<span class="go">s       category</span>
<span class="go">vals       int64</span>
<span class="go">dtype: object</span>

<span class="gp">In [143]: </span><span class="n">df2</span><span class="p">[</span><span class="s">&quot;vals&quot;</span><span class="p">]</span>
<span class="gr">Out[143]: </span>
<span class="go">0    1</span>
<span class="go">1    2</span>
<span class="go">2    3</span>
<span class="go">3    4</span>
<span class="go">4    5</span>
<span class="go">5    6</span>
<span class="go">Name: vals, dtype: int64</span>

<span class="c"># Redo the category</span>
<span class="gp">In [144]: </span><span class="n">df2</span><span class="p">[</span><span class="s">&quot;vals&quot;</span><span class="p">]</span> <span class="o">=</span> <span class="n">df2</span><span class="p">[</span><span class="s">&quot;vals&quot;</span><span class="p">]</span><span class="o">.</span><span class="n">astype</span><span class="p">(</span><span class="s">&quot;category&quot;</span><span class="p">)</span>

<span class="gp">In [145]: </span><span class="n">df2</span><span class="p">[</span><span class="s">&quot;vals&quot;</span><span class="p">]</span><span class="o">.</span><span class="n">cat</span><span class="o">.</span><span class="n">levels</span> <span class="o">=</span> <span class="nb">list</span><span class="p">(</span><span class="n">df2</span><span class="p">[</span><span class="s">&quot;vals&quot;</span><span class="p">]</span><span class="o">.</span><span class="n">cat</span><span class="o">.</span><span class="n">levels</span><span class="p">)</span> <span class="o">+</span> <span class="p">[</span><span class="s">&quot;medium&quot;</span><span class="p">,</span> <span class="s">&quot;very bad&quot;</span><span class="p">]</span>

<span class="gp">In [146]: </span><span class="n">df2</span><span class="p">[</span><span class="s">&quot;vals&quot;</span><span class="p">]</span><span class="o">.</span><span class="n">cat</span><span class="o">.</span><span class="n">reorder_levels</span><span class="p">([</span><span class="s">&quot;very bad&quot;</span><span class="p">,</span> <span class="s">&quot;bad&quot;</span><span class="p">,</span> <span class="s">&quot;medium&quot;</span><span class="p">,</span> <span class="s">&quot;good&quot;</span><span class="p">,</span> <span class="s">&quot;very good&quot;</span><span class="p">])</span>
<span class="go">---------------------------------------------------------------------------</span>
<span class="go">ValueError                                Traceback (most recent call last)</span>
<span class="go">&lt;ipython-input-146-d48b87b27d90&gt; in &lt;module&gt;()</span>
<span class="go">----&gt; 1 df2[&quot;vals&quot;].cat.reorder_levels([&quot;very bad&quot;, &quot;bad&quot;, &quot;medium&quot;, &quot;good&quot;, &quot;very good&quot;])</span>

<span class="go">c:\data\external\pandas\pandas\core\categorical.pyc in reorder_levels(self, new_levels, ordered)</span>
<span class="go">    342 </span>
<span class="go">    343         if len(new_levels) != len(self._levels):</span>
<span class="go">--&gt; 344             raise ValueError(&#39;Reordered levels must be of same length as old levels&#39;)</span>
<span class="go">    345         if len(new_levels-self._levels):</span>
<span class="go">    346             raise ValueError(&#39;Reordered levels be the same as the original levels&#39;)</span>

<span class="go">ValueError: Reordered levels must be of same length as old levels</span>

<span class="gp">In [147]: </span><span class="n">df2</span><span class="o">.</span><span class="n">dtypes</span>
<span class="gr">Out[147]: </span>
<span class="go">s       category</span>
<span class="go">vals    category</span>
<span class="go">dtype: object</span>

<span class="gp">In [148]: </span><span class="n">df2</span><span class="p">[</span><span class="s">&quot;vals&quot;</span><span class="p">]</span>
<span class="gr">Out[148]: </span>
<span class="go">0    1</span>
<span class="go">1    2</span>
<span class="go">2    3</span>
<span class="go">3    4</span>
<span class="go">4    5</span>
<span class="go">5    6</span>
<span class="go">Name: vals, dtype: category</span>
</pre></div>
</div>

</div>

<div class="section" id="missing-data">
<h2>Missing Data<a class="headerlink" href="#missing-data" title="Permalink to this headline">¶</a></h2>
<p>pandas primarily uses the value <cite>np.nan</cite> to represent missing data. It is by
default not included in computations. See the <a class="reference internal" href="missing_data.html#missing-data"><em>Missing Data section</em></a></p>
<p>There are two ways a <cite>np.nan</cite> can be represented in <cite>Categorical</cite>: either the value is not
available or <cite>np.nan</cite> is a valid level.</p>
<div class="highlight-ipython"><div class="highlight"><pre><span class="gp">In [149]: </span><span class="n">s</span> <span class="o">=</span> <span class="n">pd</span><span class="o">.</span><span class="n">Series</span><span class="p">(</span><span class="n">pd</span><span class="o">.</span><span class="n">Categorical</span><span class="p">([</span><span class="s">&quot;a&quot;</span><span class="p">,</span><span class="s">&quot;b&quot;</span><span class="p">,</span><span class="n">np</span><span class="o">.</span><span class="n">nan</span><span class="p">,</span><span class="s">&quot;a&quot;</span><span class="p">]))</span>

<span class="gp">In [150]: </span><span class="n">s</span>
<span class="gr">Out[150]: </span>
<span class="go">0      a</span>
<span class="go">1      b</span>
<span class="go">2    NaN</span>
<span class="go">3      a</span>
<span class="go">dtype: category</span>

<span class="c"># only two levels</span>
<span class="gp">In [151]: </span><span class="n">s</span><span class="o">.</span><span class="n">cat</span><span class="o">.</span><span class="n">levels</span>
<span class="gr">Out[151]: </span><span class="n">Index</span><span class="p">([</span><span class="s">u&#39;a&#39;</span><span class="p">,</span> <span class="s">u&#39;b&#39;</span><span class="p">],</span> <span class="n">dtype</span><span class="o">=</span><span class="s">&#39;object&#39;</span><span class="p">)</span>

<span class="gp">In [152]: </span><span class="n">s2</span> <span class="o">=</span> <span class="n">pd</span><span class="o">.</span><span class="n">Series</span><span class="p">(</span><span class="n">pd</span><span class="o">.</span><span class="n">Categorical</span><span class="p">([</span><span class="s">&quot;a&quot;</span><span class="p">,</span><span class="s">&quot;b&quot;</span><span class="p">,</span><span class="s">&quot;c&quot;</span><span class="p">,</span><span class="s">&quot;a&quot;</span><span class="p">]))</span>

<span class="gp">In [153]: </span><span class="n">s2</span><span class="o">.</span><span class="n">cat</span><span class="o">.</span><span class="n">levels</span> <span class="o">=</span> <span class="p">[</span><span class="mi">1</span><span class="p">,</span><span class="mi">2</span><span class="p">,</span><span class="n">np</span><span class="o">.</span><span class="n">nan</span><span class="p">]</span>

<span class="gp">In [154]: </span><span class="n">s2</span>
<span class="gr">Out[154]: </span>
<span class="go">0     1</span>
<span class="go">1     2</span>
<span class="go">2   NaN</span>
<span class="go">3     1</span>
<span class="go">dtype: category</span>

<span class="c"># three levels, np.nan included</span>
<span class="c"># Note: as int arrays can&#39;t hold NaN the levels were converted to float</span>
<span class="gp">In [155]: </span><span class="n">s2</span><span class="o">.</span><span class="n">cat</span><span class="o">.</span><span class="n">levels</span>
<span class="gr">Out[155]: </span><span class="n">Float64Index</span><span class="p">([</span><span class="mf">1.0</span><span class="p">,</span> <span class="mf">2.0</span><span class="p">,</span> <span class="n">nan</span><span class="p">],</span> <span class="n">dtype</span><span class="o">=</span><span class="s">&#39;float64&#39;</span><span class="p">)</span>
</pre></div>
</div>

</div>

<div class="section" id="gotchas">
<h2>Gotchas<a class="headerlink" href="#gotchas" title="Permalink to this headline">¶</a></h2>
<div class="section" id="categorical-is-not-a-numpy-array">
<h3><cite>Categorical</cite> is not a <cite>numpy</cite> array<a class="headerlink" href="#categorical-is-not-a-numpy-array" title="Permalink to this headline">¶</a></h3>
<p>Currently, <cite>Categorical</cite> and the corresponding <tt class="docutils literal"><span class="pre">category</span></tt> <cite>Series</cite> is implemented as a python
object and not as a low level <cite>numpy</cite> array dtype. This leads to some problems.</p>
<p><cite>numpy</cite> itself doesn&#8217;t know about the new <cite>dtype</cite>:</p>
<div class="highlight-ipython"><div class="highlight"><pre><span class="gp">In [156]: </span><span class="k">try</span><span class="p">:</span>
<span class="gp">   .....:</span>     <span class="n">np</span><span class="o">.</span><span class="n">dtype</span><span class="p">(</span><span class="s">&quot;category&quot;</span><span class="p">)</span>
<span class="gp">   .....:</span> <span class="k">except</span> <span class="ne">TypeError</span> <span class="k">as</span> <span class="n">e</span><span class="p">:</span>
<span class="gp">   .....:</span>      <span class="k">print</span><span class="p">(</span><span class="s">&quot;TypeError: &quot;</span> <span class="o">+</span> <span class="nb">str</span><span class="p">(</span><span class="n">e</span><span class="p">))</span>
<span class="gp">   .....:</span> 
<span class="go">TypeError: data type &quot;category&quot; not understood</span>

<span class="gp">In [157]: </span><span class="n">dtype</span> <span class="o">=</span> <span class="n">pd</span><span class="o">.</span><span class="n">Categorical</span><span class="p">([</span><span class="s">&quot;a&quot;</span><span class="p">])</span><span class="o">.</span><span class="n">dtype</span>

<span class="gp">In [158]: </span><span class="k">try</span><span class="p">:</span>
<span class="gp">   .....:</span>     <span class="n">np</span><span class="o">.</span><span class="n">dtype</span><span class="p">(</span><span class="n">dtype</span><span class="p">)</span>
<span class="gp">   .....:</span> <span class="k">except</span> <span class="ne">TypeError</span> <span class="k">as</span> <span class="n">e</span><span class="p">:</span>
<span class="gp">   .....:</span>      <span class="k">print</span><span class="p">(</span><span class="s">&quot;TypeError: &quot;</span> <span class="o">+</span> <span class="nb">str</span><span class="p">(</span><span class="n">e</span><span class="p">))</span>
<span class="gp">   .....:</span> 
<span class="go">TypeError: data type not understood</span>

<span class="c"># dtype comparisons work:</span>
<span class="gp">In [159]: </span><span class="n">dtype</span> <span class="o">==</span> <span class="n">np</span><span class="o">.</span><span class="n">str_</span>
<span class="gr">Out[159]: </span><span class="bp">False</span>

<span class="gp">In [160]: </span><span class="n">np</span><span class="o">.</span><span class="n">str_</span> <span class="o">==</span> <span class="n">dtype</span>
<span class="gr">Out[160]: </span><span class="bp">False</span>
</pre></div>
</div>

<p>Using <cite>numpy</cite> functions on a <cite>Series</cite> of type <tt class="docutils literal"><span class="pre">category</span></tt> should not work as <cite>Categoricals</cite>
are not numeric data (even in the case that <tt class="docutils literal"><span class="pre">.levels</span></tt> is numeric).</p>

<div class="highlight-ipython"><div class="highlight"><pre><span class="gp">In [161]: </span><span class="n">s</span> <span class="o">=</span> <span class="n">pd</span><span class="o">.</span><span class="n">Series</span><span class="p">(</span><span class="n">pd</span><span class="o">.</span><span class="n">Categorical</span><span class="p">([</span><span class="mi">1</span><span class="p">,</span><span class="mi">2</span><span class="p">,</span><span class="mi">3</span><span class="p">,</span><span class="mi">4</span><span class="p">]))</span>

<span class="gp">In [162]: </span><span class="k">try</span><span class="p">:</span>
<span class="gp">   .....:</span>     <span class="n">np</span><span class="o">.</span><span class="n">sum</span><span class="p">(</span><span class="n">s</span><span class="p">)</span>
<span class="gp">   .....:</span> <span class="k">except</span> <span class="ne">TypeError</span> <span class="k">as</span> <span class="n">e</span><span class="p">:</span>
<span class="gp">   .....:</span>      <span class="k">print</span><span class="p">(</span><span class="s">&quot;TypeError: &quot;</span> <span class="o">+</span> <span class="nb">str</span><span class="p">(</span><span class="n">e</span><span class="p">))</span>
<span class="gp">   .....:</span> 
<span class="go">TypeError: Categorical cannot perform the operation sum</span>
</pre></div>
</div>

<div class="admonition note">
<p class="first admonition-title">Note</p>
<p class="last">If such a function works, please file a bug at <a class="reference external" href="https://github.com/pydata/pandas">https://github.com/pydata/pandas</a>!</p>
</div>

</div>

<div class="section" id="side-effects">
<h3>Side effects<a class="headerlink" href="#side-effects" title="Permalink to this headline">¶</a></h3>
<p>Constructing a <cite>Series</cite> from a <cite>Categorical</cite> will not copy the input <cite>Categorical</cite>. This
means that changes to the <cite>Series</cite> will in most cases change the original <cite>Categorical</cite>:</p>
<div class="highlight-ipython"><div class="highlight"><pre><span class="gp">In [163]: </span><span class="n">cat</span> <span class="o">=</span> <span class="n">pd</span><span class="o">.</span><span class="n">Categorical</span><span class="p">([</span><span class="mi">1</span><span class="p">,</span><span class="mi">2</span><span class="p">,</span><span class="mi">3</span><span class="p">,</span><span class="mi">10</span><span class="p">],</span> <span class="n">levels</span><span class="o">=</span><span class="p">[</span><span class="mi">1</span><span class="p">,</span><span class="mi">2</span><span class="p">,</span><span class="mi">3</span><span class="p">,</span><span class="mi">4</span><span class="p">,</span><span class="mi">10</span><span class="p">])</span>

<span class="gp">In [164]: </span><span class="n">s</span> <span class="o">=</span> <span class="n">pd</span><span class="o">.</span><span class="n">Series</span><span class="p">(</span><span class="n">cat</span><span class="p">,</span> <span class="n">name</span><span class="o">=</span><span class="s">&quot;cat&quot;</span><span class="p">)</span>

<span class="gp">In [165]: </span><span class="n">cat</span>
<span class="gr">Out[165]: </span>
<span class="go">  1</span>
<span class="go">  2</span>
<span class="go">  3</span>
<span class="go"> 10</span>
<span class="go">Levels (5): Int64Index([ 1,  2,  3,  4, 10], dtype=int64), ordered</span>

<span class="gp">In [166]: </span><span class="n">s</span><span class="o">.</span><span class="n">iloc</span><span class="p">[</span><span class="mi">0</span><span class="p">:</span><span class="mi">2</span><span class="p">]</span> <span class="o">=</span> <span class="mi">10</span>

<span class="gp">In [167]: </span><span class="n">cat</span>
<span class="gr">Out[167]: </span>
<span class="go"> 10</span>
<span class="go"> 10</span>
<span class="go">  3</span>
<span class="go"> 10</span>
<span class="go">Levels (5): Int64Index([ 1,  2,  3,  4, 10], dtype=int64), ordered</span>

<span class="gp">In [168]: </span><span class="n">df</span> <span class="o">=</span> <span class="n">pd</span><span class="o">.</span><span class="n">DataFrame</span><span class="p">(</span><span class="n">s</span><span class="p">)</span>

<span class="gp">In [169]: </span><span class="n">df</span><span class="p">[</span><span class="s">&quot;cat&quot;</span><span class="p">]</span><span class="o">.</span><span class="n">cat</span><span class="o">.</span><span class="n">levels</span> <span class="o">=</span> <span class="p">[</span><span class="mi">1</span><span class="p">,</span><span class="mi">2</span><span class="p">,</span><span class="mi">3</span><span class="p">,</span><span class="mi">4</span><span class="p">,</span><span class="mi">5</span><span class="p">]</span>

<span class="gp">In [170]: </span><span class="n">cat</span>
<span class="gr">Out[170]: </span>
<span class="go"> 5</span>
<span class="go"> 5</span>
<span class="go"> 3</span>
<span class="go"> 5</span>
<span class="go">Levels (5): Int64Index([1, 2, 3, 4, 5], dtype=int64), ordered</span>
</pre></div>
</div>

<p>Use <tt class="docutils literal"><span class="pre">copy=True</span></tt> to prevent such a behaviour:</p>

<div class="highlight-ipython"><div class="highlight"><pre><span class="gp">In [171]: </span><span class="n">cat</span> <span class="o">=</span> <span class="n">pd</span><span class="o">.</span><span class="n">Categorical</span><span class="p">([</span><span class="mi">1</span><span class="p">,</span><span class="mi">2</span><span class="p">,</span><span class="mi">3</span><span class="p">,</span><span class="mi">10</span><span class="p">],</span> <span class="n">levels</span><span class="o">=</span><span class="p">[</span><span class="mi">1</span><span class="p">,</span><span class="mi">2</span><span class="p">,</span><span class="mi">3</span><span class="p">,</span><span class="mi">4</span><span class="p">,</span><span class="mi">10</span><span class="p">])</span>

<span class="gp">In [172]: </span><span class="n">s</span> <span class="o">=</span> <span class="n">pd</span><span class="o">.</span><span class="n">Series</span><span class="p">(</span><span class="n">cat</span><span class="p">,</span> <span class="n">name</span><span class="o">=</span><span class="s">&quot;cat&quot;</span><span class="p">,</span> <span class="n">copy</span><span class="o">=</span><span class="bp">True</span><span class="p">)</span>

<span class="gp">In [173]: </span><span class="n">cat</span>
<span class="gr">Out[173]: </span>
<span class="go">  1</span>
<span class="go">  2</span>
<span class="go">  3</span>
<span class="go"> 10</span>
<span class="go">Levels (5): Int64Index([ 1,  2,  3,  4, 10], dtype=int64), ordered</span>

<span class="gp">In [174]: </span><span class="n">s</span><span class="o">.</span><span class="n">iloc</span><span class="p">[</span><span class="mi">0</span><span class="p">:</span><span class="mi">2</span><span class="p">]</span> <span class="o">=</span> <span class="mi">10</span>

<span class="gp">In [175]: </span><span class="n">cat</span>
<span class="gr">Out[175]: </span>
<span class="go">  1</span>
<span class="go">  2</span>
<span class="go">  3</span>
<span class="go"> 10</span>
<span class="go">Levels (5): Int64Index([ 1,  2,  3,  4, 10], dtype=int64), ordered</span>
</pre></div>
</div>

<div class="admonition note">
<p class="first admonition-title">Note</p>
<p class="last">This also happens in some cases when you supply a <cite>numpy</cite> array instea dof a <cite>Categorical</cite>:
using an int array (e.g. <tt class="docutils literal"><span class="pre">np.array([1,2,3,4])</span></tt>) will exhibit the same behaviour, but using
a string array (e.g. <tt class="docutils literal"><span class="pre">np.array([&quot;a&quot;,&quot;b&quot;,&quot;c&quot;,&quot;a&quot;])</span></tt>) will not.</p>
</div>

</div>

<div class="section" id="danger-of-confusion">
<h3>Danger of confusion<a class="headerlink" href="#danger-of-confusion" title="Permalink to this headline">¶</a></h3>
<p>Both <cite>Series</cite> and <cite>Categorical</cite> have a method <tt class="docutils literal"><span class="pre">.reorder_levels()</span></tt> but for different things. For
Series of type <tt class="docutils literal"><span class="pre">category</span></tt> this means that there is some danger to confuse both methods.</p>
<div class="highlight-ipython"><div class="highlight"><pre><span class="gp">In [176]: </span><span class="n">s</span> <span class="o">=</span> <span class="n">pd</span><span class="o">.</span><span class="n">Series</span><span class="p">(</span><span class="n">pd</span><span class="o">.</span><span class="n">Categorical</span><span class="p">([</span><span class="mi">1</span><span class="p">,</span><span class="mi">2</span><span class="p">,</span><span class="mi">3</span><span class="p">,</span><span class="mi">4</span><span class="p">]))</span>

<span class="gp">In [177]: </span><span class="k">print</span><span class="p">(</span><span class="n">s</span><span class="o">.</span><span class="n">cat</span><span class="o">.</span><span class="n">levels</span><span class="p">)</span>
<span class="go">Int64Index([1, 2, 3, 4], dtype=&#39;int64&#39;)</span>

<span class="c"># wrong and raises an error:</span>
<span class="gp">In [178]: </span><span class="k">try</span><span class="p">:</span>
<span class="gp">   .....:</span>     <span class="n">s</span><span class="o">.</span><span class="n">reorder_levels</span><span class="p">([</span><span class="mi">4</span><span class="p">,</span><span class="mi">3</span><span class="p">,</span><span class="mi">2</span><span class="p">,</span><span class="mi">1</span><span class="p">])</span>
<span class="gp">   .....:</span> <span class="k">except</span> <span class="ne">Exception</span> <span class="k">as</span> <span class="n">e</span><span class="p">:</span>
<span class="gp">   .....:</span>     <span class="k">print</span><span class="p">(</span><span class="s">&quot;Exception: &quot;</span> <span class="o">+</span> <span class="nb">str</span><span class="p">(</span><span class="n">e</span><span class="p">))</span>
<span class="gp">   .....:</span> 
<span class="go">Exception: Can only reorder levels on a hierarchical axis.</span>

<span class="c"># right</span>
<span class="gp">In [179]: </span><span class="n">s</span><span class="o">.</span><span class="n">cat</span><span class="o">.</span><span class="n">reorder_levels</span><span class="p">([</span><span class="mi">4</span><span class="p">,</span><span class="mi">3</span><span class="p">,</span><span class="mi">2</span><span class="p">,</span><span class="mi">1</span><span class="p">])</span>

<span class="gp">In [180]: </span><span class="k">print</span><span class="p">(</span><span class="n">s</span><span class="o">.</span><span class="n">cat</span><span class="o">.</span><span class="n">levels</span><span class="p">)</span>
<span class="go">Int64Index([4, 3, 2, 1], dtype=&#39;int64&#39;)</span>
</pre></div>
</div>

<p>See also the API documentation for <a class="reference internal" href="generated/pandas.Series.reorder_levels.html#pandas.Series.reorder_levels" title="pandas.Series.reorder_levels"><tt class="xref py py-func docutils literal"><span class="pre">pandas.Series.reorder_levels()</span></tt></a> and
<tt class="xref py py-func docutils literal"><span class="pre">pandas.Categorical.reorder_levels()</span></tt></p>

</div>

<div class="section" id="old-style-constructor-usage">
<h3>Old style constructor usage<a class="headerlink" href="#old-style-constructor-usage" title="Permalink to this headline">¶</a></h3>
<p>I earlier versions, a <cite>Categorical</cite> could be constructed by passing in precomputed <cite>level_codes</cite>
(called then <cite>labels</cite>) instead of values with levels. The <cite>level_codes</cite> are interpreted as pointers
to the levels with <cite>-1</cite> as <cite>NaN</cite>. This usage is now deprecated and not available unless
<tt class="docutils literal"><span class="pre">compat=True</span></tt> is passed to the constructor of <cite>Categorical</cite>.</p>
<div class="highlight-ipython"><div class="highlight"><pre><span class="gp">In [181]: </span><span class="n">cat</span> <span class="o">=</span> <span class="n">pd</span><span class="o">.</span><span class="n">Categorical</span><span class="p">([</span><span class="mi">1</span><span class="p">,</span><span class="mi">2</span><span class="p">],</span> <span class="n">levels</span><span class="o">=</span><span class="p">[</span><span class="mi">1</span><span class="p">,</span><span class="mi">2</span><span class="p">,</span><span class="mi">3</span><span class="p">],</span> <span class="n">compat</span><span class="o">=</span><span class="bp">True</span><span class="p">)</span>

<span class="gp">In [182]: </span><span class="n">cat</span><span class="o">.</span><span class="n">get_values</span><span class="p">()</span>
<span class="gr">Out[182]: </span><span class="n">array</span><span class="p">([</span><span class="mi">2</span><span class="p">,</span> <span class="mi">3</span><span class="p">],</span> <span class="n">dtype</span><span class="o">=</span><span class="n">int64</span><span class="p">)</span>
</pre></div>
</div>

<p>In the default case (<tt class="docutils literal"><span class="pre">compat=False</span></tt>) the first argument is interpreted as values.</p>

<div class="highlight-ipython"><div class="highlight"><pre><span class="gp">In [183]: </span><span class="n">cat</span> <span class="o">=</span> <span class="n">pd</span><span class="o">.</span><span class="n">Categorical</span><span class="p">([</span><span class="mi">1</span><span class="p">,</span><span class="mi">2</span><span class="p">],</span> <span class="n">levels</span><span class="o">=</span><span class="p">[</span><span class="mi">1</span><span class="p">,</span><span class="mi">2</span><span class="p">,</span><span class="mi">3</span><span class="p">],</span> <span class="n">compat</span><span class="o">=</span><span class="bp">False</span><span class="p">)</span>

<span class="gp">In [184]: </span><span class="n">cat</span><span class="o">.</span><span class="n">get_values</span><span class="p">()</span>
<span class="gr">Out[184]: </span><span class="n">array</span><span class="p">([</span><span class="mi">1</span><span class="p">,</span> <span class="mi">2</span><span class="p">],</span> <span class="n">dtype</span><span class="o">=</span><span class="n">int64</span><span class="p">)</span>
</pre></div>
</div>

<div class="admonition warning">
<p class="first admonition-title">Warning</p>
<p class="last">Using Categorical with precomputed level_codes and levels is deprecated and a <cite>FutureWarning</cite>
is raised. Please change your code to use one of the proper constructor modes instead of
adding <tt class="docutils literal"><span class="pre">compat=False</span></tt>.</p>
</div>

</div>

<div class="section" id="no-categorical-index">
<h3>No categorical index<a class="headerlink" href="#no-categorical-index" title="Permalink to this headline">¶</a></h3>
<p>There is currently no index of type <tt class="docutils literal"><span class="pre">category</span></tt>, so setting the index to a <cite>Categorical</cite> will
convert the <cite>Categorical</cite> to a normal <cite>numpy</cite> array first and therefore remove any custom
ordering of the levels:</p>
<div class="highlight-ipython"><div class="highlight"><pre><span class="gp">In [185]: </span><span class="n">cats</span> <span class="o">=</span> <span class="n">pd</span><span class="o">.</span><span class="n">Categorical</span><span class="p">([</span><span class="mi">1</span><span class="p">,</span><span class="mi">2</span><span class="p">,</span><span class="mi">3</span><span class="p">,</span><span class="mi">4</span><span class="p">],</span> <span class="n">levels</span><span class="o">=</span><span class="p">[</span><span class="mi">4</span><span class="p">,</span><span class="mi">2</span><span class="p">,</span><span class="mi">3</span><span class="p">,</span><span class="mi">1</span><span class="p">])</span>

<span class="gp">In [186]: </span><span class="n">strings</span> <span class="o">=</span> <span class="p">[</span><span class="s">&quot;a&quot;</span><span class="p">,</span><span class="s">&quot;b&quot;</span><span class="p">,</span><span class="s">&quot;c&quot;</span><span class="p">,</span><span class="s">&quot;d&quot;</span><span class="p">]</span>

<span class="gp">In [187]: </span><span class="n">values</span> <span class="o">=</span> <span class="p">[</span><span class="mi">4</span><span class="p">,</span><span class="mi">2</span><span class="p">,</span><span class="mi">3</span><span class="p">,</span><span class="mi">1</span><span class="p">]</span>

<span class="gp">In [188]: </span><span class="n">df</span> <span class="o">=</span> <span class="n">pd</span><span class="o">.</span><span class="n">DataFrame</span><span class="p">({</span><span class="s">&quot;strings&quot;</span><span class="p">:</span><span class="n">strings</span><span class="p">,</span> <span class="s">&quot;values&quot;</span><span class="p">:</span><span class="n">values</span><span class="p">},</span> <span class="n">index</span><span class="o">=</span><span class="n">cats</span><span class="p">)</span>

<span class="gp">In [189]: </span><span class="n">df</span><span class="o">.</span><span class="n">index</span>
<span class="gr">Out[189]: </span><span class="n">Int64Index</span><span class="p">([</span><span class="mi">1</span><span class="p">,</span> <span class="mi">2</span><span class="p">,</span> <span class="mi">3</span><span class="p">,</span> <span class="mi">4</span><span class="p">],</span> <span class="n">dtype</span><span class="o">=</span><span class="s">&#39;int64&#39;</span><span class="p">)</span>

<span class="c"># This should sort by levels but does not as there is no CategoricalIndex!</span>
<span class="gp">In [190]: </span><span class="n">df</span><span class="o">.</span><span class="n">sort_index</span><span class="p">()</span>
<span class="gr">Out[190]: </span>
<span class="go">  strings  values</span>
<span class="go">1       a       4</span>
<span class="go">2       b       2</span>
<span class="go">3       c       3</span>
<span class="go">4       d       1</span>
</pre></div>
</div>

<div class="admonition note">
<p class="first admonition-title">Note</p>
<p class="last">This could change if a <cite>CategoricalIndex</cite> is implemented (see
<a class="reference external" href="https://github.com/pydata/pandas/issues/7629">https://github.com/pydata/pandas/issues/7629</a>)</p>
</div>

</div>

<div class="section" id="dtype-in-apply">
<h3>dtype in apply<a class="headerlink" href="#dtype-in-apply" title="Permalink to this headline">¶</a></h3>
<p>Pandas currently does not preserve the dtype in apply functions: If you apply along rows you get
a <cite>Series</cite> of <tt class="docutils literal"><span class="pre">object</span></tt> <cite>dtype</cite> (same as getting a row -&gt; getting one element will return a
basic type) and applying along columns will also convert to object.</p>
<div class="highlight-ipython"><div class="highlight"><pre><span class="gp">In [191]: </span><span class="n">df</span> <span class="o">=</span> <span class="n">pd</span><span class="o">.</span><span class="n">DataFrame</span><span class="p">({</span><span class="s">&quot;a&quot;</span><span class="p">:[</span><span class="mi">1</span><span class="p">,</span><span class="mi">2</span><span class="p">,</span><span class="mi">3</span><span class="p">,</span><span class="mi">4</span><span class="p">],</span> <span class="s">&quot;b&quot;</span><span class="p">:[</span><span class="s">&quot;a&quot;</span><span class="p">,</span><span class="s">&quot;b&quot;</span><span class="p">,</span><span class="s">&quot;c&quot;</span><span class="p">,</span><span class="s">&quot;d&quot;</span><span class="p">],</span> <span class="s">&quot;cats&quot;</span><span class="p">:</span><span class="n">pd</span><span class="o">.</span><span class="n">Categorical</span><span class="p">([</span><span class="mi">1</span><span class="p">,</span><span class="mi">2</span><span class="p">,</span><span class="mi">3</span><span class="p">,</span><span class="mi">2</span><span class="p">])})</span>

<span class="gp">In [192]: </span><span class="n">df</span><span class="o">.</span><span class="n">apply</span><span class="p">(</span><span class="k">lambda</span> <span class="n">row</span><span class="p">:</span> <span class="nb">type</span><span class="p">(</span><span class="n">row</span><span class="p">[</span><span class="s">&quot;cats&quot;</span><span class="p">]),</span> <span class="n">axis</span><span class="o">=</span><span class="mi">1</span><span class="p">)</span>
<span class="gr">Out[192]: </span>
<span class="go">0    &lt;type &#39;long&#39;&gt;</span>
<span class="go">1    &lt;type &#39;long&#39;&gt;</span>
<span class="go">2    &lt;type &#39;long&#39;&gt;</span>
<span class="go">3    &lt;type &#39;long&#39;&gt;</span>
<span class="go">dtype: object</span>

<span class="gp">In [193]: </span><span class="n">df</span><span class="o">.</span><span class="n">apply</span><span class="p">(</span><span class="k">lambda</span> <span class="n">col</span><span class="p">:</span> <span class="n">col</span><span class="o">.</span><span class="n">dtype</span><span class="p">,</span> <span class="n">axis</span><span class="o">=</span><span class="mi">0</span><span class="p">)</span>
<span class="gr">Out[193]: </span>
<span class="go">a       object</span>
<span class="go">b       object</span>
<span class="go">cats    object</span>
<span class="go">dtype: object</span>
</pre></div>
</div>

</div>

<div class="section" id="future-compatibility">
<h3>Future compatibility<a class="headerlink" href="#future-compatibility" title="Permalink to this headline">¶</a></h3>
<p>As <cite>Categorical</cite> is not a native <cite>numpy</cite> dtype, the implementation details of
<cite>Series.cat</cite> can change if such a <cite>numpy</cite> dtype is implemented.</p>
</div>

</div>
</div>


Labels:
name: Docs
name: Categorical
description: Categorical Data Type

Comments:
can u show the rendered one in the top section?

I am not a big fan of the long Series  accessors,  e.g. `category_levels`

however, easy enough to do something like

`Series.cat.levels` or `Series.category.levels`  (or `Series.values.levels`, which u can do now)

**EDIT**: Doc removed and new version added below

@jreback: I like the idea with `Series.cat`. What should `Series.cat` do when the series is not a `category`? Same as `Series.str` (-> convert) or  raise an `TypeError`?

naw will raise a TypeError

very easy impl

```
@property
def cat(self):
    if self.dtype != 'category':
       raise TypeError('not a category')
    return self.values

```

I've updated the docs and added new unittests for slicing (lots of failures and crashes :-( )

Edit: I also changed a typo in one of my earlier commits

I updated the main PR; all of the slicing FIXME's are done: https://github.com/pydata/pandas/pull/7217

Thanks! I started to fix that at the completely wrong place and got stuck right away :/

@jreback I've fixed some screwups in my slicing testcases and added more slicing tests.

I also added assigning testcases (which of course fail because assigning is not yet implmented). I do hope that I got the testcases right...

ok thanks

Assign value already in levels

```
> df = orig.copy()
(Pdb) n
> /mnt/home/jreback/pandas/pandas/tests/test_categorical.py(979)test_assigning_ops()
-> df.iloc[2,0] = "b"
(Pdb) p df
  cats  values
h    a       1
i    a       1
j    b       1
k    a       1
l    a       1
m    a       1
n    a       1
```

Assign value NOT in levels, just add the levels to the end
(or is this a Value Error?)

```
(Pdb) !df.iloc[2,0]
'b'
(Pdb) !df.iloc[2,0] = 'c'
(Pdb) p df
  cats  values
h    a       1
i    a       1
j    c       1
k    a       1
l    a       1
m    a       1
n    a       1
(Pdb) p df['cats'].cat.levels
Index([u'a', u'b', u'c'], dtype='object')
(Pdb) p df['cats'].cat.labels
array([0, 0, 2, 0, 0, 0, 0])
(Pdb) 
```

Assigning values not in levels should raise: if you want to have them included, you need to first add such a level and then assign the value.

If levels and level ordering has a meaning, it makes no sense to add a level just because a value was assigned: `s.levels = ["best", "good", "medium", "bad", "worst"]; s.ordered=True`. Now assigning a `milk` (or a float/int) makes no sense.

ok gr8....thxs

This is what R does:

``` R
> factor(c(1,2,3,4))
[1] 1 2 3 4
Levels: 1 2 3 4
> a = factor(c(1,2,3,4))
> a[1]
[1] 1
Levels: 1 2 3 4
> a[1] <- 6
Warning message:
In `[<-.factor`(`*tmp*`, 1, value = 6) : invalid factor level, NA generated
```

(One difference: a single value is also of type factor...)

ok....changed to raising (easy enough for say an `ordered=False` to allow this, just append to the level set)

yep...that's what I do from your branch

[removed and added in the first comment]

I changed `newstyle` -> `compat` (as that's the usual argument). So one can set it if its an issue to preserver old behavior. good

But this would mean that the default is changed to "new style", which will break old code?

I think I changed the semantics, but were'nt you +1 on just making newstyle (compat=False) the default anyhow?

I'm fine with it, but wasn't sure if you intended to break old code with a  rename :-)

If you are ok with the API change, I would remove the codepath completely and just add something similar to the `fastpath=True` case.

ok....i'll leave for now, pls review after I push again (i'll let u know)

@JanSchulz 

got fillna working with a value (assume its an error if its NOT in the levels)

for method='pad/bfill` its an error if its not an ordered Categorical
then I guess its normal propogation yes?

As far as I understood `pad` it just takes the value at position x-1, so I think it should also work for unordered categoricals. `[1,2,nan,3] -> [1,2,2,3]` because `2` was at the second position? 

it will work, but I think its wrong (trivial to allow it). I'll leave it in with a FIXME and you can take a look.

What's your reason for "wrong"?

hmm...I guess they DO have an ordering (which is essentially in index order), 

ok then

ok, new commits are up (I didn't fix the concat stuff yet though)

take a look at the `compat` and `fillna` issues as discussed

@JanSchulz pls create another issue for 0.15 for future changes, e.g. `to_csv`/`to_hdf` and anything else non-essential.